2024-04-26T20:11:05.265 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T20:11:05.265 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rishabh@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} --archive /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159 --name rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --verbose -- /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159/orig.config.yaml 2024-04-26T20:11:05.322 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T20:11:05.404 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159 branch: wip-rishabh-testing-20240426.111959 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} email: null first_in_suite: false job_id: '7675159' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWvmtzbHbLFhJ97Cu/iv+xsO6C9RwzQEyhw12pI0j8348FADsk9O6WHmEVYqNJ5KG/VCJef3q9XUBl6PWeMwr8= smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMIaKD/jMdpOkfA1Ox+h+n8f6ujZhyxeF1Lrd4qG4U4yuoyGiSglOH20N5j+stq60Rkw8S3lkxe2EB3+742n4kQ= smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB5kOYp3W0wKwhvvFJ9sier57RU5hsuezJTJFqoHJPZt/gvPOVEs2SocAoB7qdgTKvhqTu8Q2kpF0xie1BU28pc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/fsstress.sh timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_19:30:57 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T20:11:05.404 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-26T20:11:05.405 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-26T20:11:05.405 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T20:11:05.407 INFO:teuthology.task.internal:Checking packages... 2024-04-26T20:11:05.428 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-26T20:11:05.428 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T20:11:05.428 INFO:teuthology.packaging:ref: None 2024-04-26T20:11:05.428 INFO:teuthology.packaging:tag: None 2024-04-26T20:11:05.428 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-26T20:11:05.428 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:11:05.428 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-26T20:11:05.569 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-26T20:11:05.570 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T20:11:05.585 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T20:11:05.585 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T20:11:05.641 INFO:teuthology.task.internal:Saving configuration 2024-04-26T20:11:05.660 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T20:11:05.667 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T20:11:05.691 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 20:06:03.127248', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWvmtzbHbLFhJ97Cu/iv+xsO6C9RwzQEyhw12pI0j8348FADsk9O6WHmEVYqNJ5KG/VCJef3q9XUBl6PWeMwr8='} 2024-04-26T20:11:05.711 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 20:06:03.130119', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMIaKD/jMdpOkfA1Ox+h+n8f6ujZhyxeF1Lrd4qG4U4yuoyGiSglOH20N5j+stq60Rkw8S3lkxe2EB3+742n4kQ='} 2024-04-26T20:11:05.734 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 20:06:03.128825', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB5kOYp3W0wKwhvvFJ9sier57RU5hsuezJTJFqoHJPZt/gvPOVEs2SocAoB7qdgTKvhqTu8Q2kpF0xie1BU28pc='} 2024-04-26T20:11:05.734 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T20:11:05.738 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T20:11:05.738 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T20:11:05.738 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T20:11:05.738 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T20:11:05.833 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcc160e5160>, signals=[15]) 2024-04-26T20:11:05.833 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T20:11:05.840 INFO:teuthology.task.internal:Opening connections... 2024-04-26T20:11:05.840 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-04-26T20:11:05.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T20:11:05.919 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-04-26T20:11:05.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T20:11:05.992 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2024-04-26T20:11:05.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T20:11:06.064 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T20:11:06.069 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-04-26T20:11:06.086 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-04-26T20:11:06.087 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-04-26T20:11:06.144 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T20:11:06.145 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T20:11:06.145 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T20:11:06.145 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-04-26T20:11:06.165 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-04-26T20:11:06.182 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-04-26T20:11:06.183 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-04-26T20:11:06.239 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="9" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="9" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el9" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:LOGO="fedora-logo-icon" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T20:11:06.240 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T20:11:06.241 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T20:11:06.241 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-04-26T20:11:06.264 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2024-04-26T20:11:06.282 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2024-04-26T20:11:06.282 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2024-04-26T20:11:06.339 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Stream" 2024-04-26T20:11:06.339 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="9" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="9" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el9" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:LOGO="fedora-logo-icon" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://centos.org/" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T20:11:06.340 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T20:11:06.341 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-04-26T20:11:06.364 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T20:11:06.374 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T20:11:06.379 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T20:11:06.379 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-04-26T20:11:06.382 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-04-26T20:11:06.384 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2024-04-26T20:11:06.400 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T20:11:06.416 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T20:11:06.416 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-04-26T20:11:06.439 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-04-26T20:11:06.441 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2024-04-26T20:11:06.553 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T20:11:06.632 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T20:11:06.647 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-26T20:11:06.647 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-26T20:11:06.647 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-26T20:11:06.647 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T20:11:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:11:06.663 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T20:11:06.820 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T20:11:06.820 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T20:11:06.821 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-04-26T20:11:06.838 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-437.el9.x86_64 2024-04-26T20:11:06.838 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T20:11:06.838 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T20:11:06.839 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum upgrade -y linux-firmware 2024-04-26T20:11:09.215 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-26T20:11:11.566 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-26T20:11:15.861 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.2 MB 00:00 2024-04-26T20:11:18.209 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-26T20:11:19.321 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-04-26T20:11:24.286 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-26T20:11:26.310 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-26T20:11:26.311 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-26T20:11:26.311 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-26T20:11:26.430 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T20:11:26.431 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T20:11:26.651 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T20:11:26.651 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T20:11:26.653 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi102.front.sepia.ceph.com to complete... 2024-04-26T20:11:38.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26 20:11:38 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T20:11:38.970 INFO:teuthology.orchestra.run.smithi102.stderr:FINISHED --2024-04-26 20:11:38-- 2024-04-26T20:11:38.970 INFO:teuthology.orchestra.run.smithi102.stderr:Total wall clock time: 12s 2024-04-26T20:11:38.971 INFO:teuthology.orchestra.run.smithi102.stderr:Downloaded: 1 files, 341M in 12s (28.5 MB/s) 2024-04-26T20:11:38.973 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T20:11:38.973 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T20:11:41.128 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying... ######################################## 2024-04-26T20:11:41.278 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing... ######################################## 2024-04-26T20:11:41.384 INFO:teuthology.orchestra.run.smithi102.stdout:Updating / installing... 2024-04-26T20:11:49.351 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T20:12:28.486 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T20:12:28.487 DEBUG:teuthology.orchestra.run.smithi102:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T20:12:28.560 INFO:teuthology.orchestra.run.smithi102.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T20:12:28.560 INFO:teuthology.orchestra.run.smithi102.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T20:12:28.560 INFO:teuthology.orchestra.run.smithi102.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T20:12:28.560 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T20:12:28.560 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T20:12:28.561 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T20:12:28.562 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T20:12:28.563 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T20:12:28.564 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T20:12:28.565 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T20:12:28.566 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T20:12:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T20:12:28.568 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T20:12:28.569 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T20:12:28.570 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T20:12:28.571 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T20:12:28.572 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T20:12:28.573 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T20:12:28.574 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T20:12:28.575 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T20:12:28.576 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T20:12:28.577 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T20:12:28.578 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T20:12:28.579 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T20:12:28.580 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T20:12:28.581 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T20:12:28.582 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T20:12:28.582 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T20:12:28.582 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T20:12:28.582 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T20:12:28.582 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T20:12:28.582 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T20:12:28.582 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T20:12:28.582 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T20:12:28.582 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T20:12:28.583 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T20:12:28.583 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T20:12:28.583 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T20:12:28.583 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T20:12:28.583 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T20:12:28.583 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T20:12:28.583 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T20:12:28.583 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T20:12:28.583 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T20:12:28.584 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T20:12:28.584 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T20:12:28.584 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T20:12:28.584 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T20:12:28.584 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T20:12:28.584 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T20:12:28.584 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T20:12:28.584 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T20:12:28.584 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T20:12:28.584 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T20:12:28.585 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T20:12:28.585 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T20:12:28.585 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T20:12:28.585 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T20:12:28.585 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T20:12:28.585 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T20:12:28.585 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T20:12:28.585 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T20:12:28.585 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T20:12:28.585 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T20:12:28.586 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T20:12:28.586 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T20:12:28.586 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T20:12:28.586 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T20:12:28.586 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T20:12:28.586 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T20:12:28.586 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T20:12:28.586 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T20:12:28.587 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T20:12:28.587 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T20:12:28.587 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T20:12:28.587 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T20:12:28.587 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T20:12:28.587 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T20:12:28.587 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T20:12:28.587 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T20:12:28.587 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T20:12:28.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T20:12:28.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T20:12:28.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T20:12:28.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T20:12:28.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T20:12:28.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T20:12:28.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T20:12:28.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T20:12:28.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T20:12:28.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T20:12:28.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T20:12:28.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T20:12:28.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T20:12:28.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T20:12:28.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T20:12:28.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T20:12:28.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T20:12:28.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T20:12:28.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T20:12:28.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T20:12:28.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T20:12:28.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T20:12:28.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T20:12:28.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T20:12:28.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T20:12:28.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T20:12:28.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T20:12:28.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T20:12:28.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T20:12:28.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T20:12:28.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T20:12:28.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T20:12:28.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T20:12:28.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T20:12:28.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T20:12:28.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T20:12:28.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T20:12:28.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T20:12:28.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T20:12:28.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T20:12:28.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T20:12:28.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T20:12:28.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T20:12:28.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T20:12:28.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T20:12:28.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T20:12:28.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T20:12:28.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T20:12:28.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T20:12:28.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T20:12:28.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T20:12:28.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T20:12:28.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T20:12:28.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T20:12:28.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T20:12:28.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T20:12:28.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T20:12:28.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T20:12:28.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T20:12:28.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T20:12:28.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T20:12:28.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T20:12:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T20:12:28.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T20:12:28.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T20:12:28.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T20:12:28.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T20:12:28.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T20:12:28.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T20:12:28.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T20:12:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T20:12:28.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T20:12:28.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T20:12:28.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T20:12:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T20:12:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T20:12:28.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T20:12:28.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T20:12:28.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T20:12:28.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T20:12:28.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T20:12:28.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T20:12:28.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T20:12:28.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T20:12:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T20:12:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T20:12:28.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T20:12:28.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T20:12:28.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T20:12:28.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T20:12:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T20:12:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T20:12:28.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T20:12:28.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T20:12:28.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T20:12:28.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T20:12:28.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T20:12:28.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T20:12:28.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T20:12:28.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T20:12:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T20:12:28.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T20:12:28.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T20:12:28.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T20:12:28.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T20:12:28.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T20:12:28.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T20:12:28.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T20:12:28.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T20:12:28.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T20:12:28.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T20:12:28.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T20:12:28.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T20:12:28.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T20:12:28.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T20:12:28.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T20:12:28.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T20:12:28.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T20:12:28.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T20:12:28.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T20:12:28.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T20:12:28.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T20:12:28.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T20:12:28.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T20:12:28.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T20:12:28.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T20:12:28.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T20:12:28.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T20:12:28.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T20:12:28.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T20:12:28.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T20:12:28.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T20:12:28.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T20:12:28.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T20:12:28.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T20:12:28.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T20:12:28.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T20:12:28.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T20:12:28.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T20:12:28.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T20:12:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T20:12:28.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T20:12:28.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T20:12:28.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T20:12:28.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T20:12:28.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T20:12:28.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T20:12:28.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T20:12:28.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T20:12:28.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T20:12:28.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T20:12:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T20:12:28.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T20:12:28.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T20:12:28.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T20:12:28.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T20:12:28.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T20:12:28.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T20:12:28.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T20:12:28.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T20:12:28.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T20:12:28.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T20:12:28.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T20:12:28.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T20:12:28.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T20:12:28.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T20:12:28.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T20:12:28.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T20:12:28.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T20:12:28.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T20:12:28.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T20:12:28.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T20:12:28.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T20:12:28.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T20:12:28.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T20:12:28.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T20:12:28.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T20:12:28.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T20:12:28.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T20:12:28.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T20:12:28.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T20:12:28.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T20:12:28.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T20:12:28.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T20:12:28.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T20:12:28.702 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T20:12:28.702 DEBUG:teuthology.orchestra.run.smithi102:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T20:12:28.726 INFO:teuthology.orchestra.run.smithi102.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T20:12:28.726 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T20:12:28.726 INFO:teuthology.orchestra.run.smithi102.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T20:12:28.726 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-04-26T20:12:28.726 INFO:teuthology.orchestra.run.smithi102.stdout:for file in vmlinuz System.map config; do 2024-04-26T20:12:28.726 INFO:teuthology.orchestra.run.smithi102.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T20:12:28.726 INFO:teuthology.orchestra.run.smithi102.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T20:12:28.726 INFO:teuthology.orchestra.run.smithi102.stdout: fi 2024-04-26T20:12:28.726 INFO:teuthology.orchestra.run.smithi102.stdout:done 2024-04-26T20:12:28.726 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T20:12:28.727 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-04-26T20:12:28.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2024-04-26T20:12:28.801 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2024-04-26T20:12:28.801 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2024-04-26T20:12:28.801 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.06 2024-04-26T20:12:28.801 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 68.el9 2024-04-26T20:12:28.801 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2024-04-26T20:12:28.801 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:Group : Unspecified 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 8264254 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : builder@centos.org 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:12:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T20:12:28.803 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T20:12:28.803 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T20:12:28.803 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2024-04-26T20:12:28.803 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:12:28.803 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T20:12:28.804 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T20:12:28.804 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T20:12:28.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T20:12:29.368 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-04-26T20:12:30.769 INFO:teuthology.orchestra.run.smithi102.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T20:12:30.802 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-04-26T20:12:30.804 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-04-26T20:12:30.823 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.pIGtR0tTzq 2024-04-26T20:12:30.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pIGtR0tTzq 2024-04-26T20:12:30.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.pIGtR0tTzq 2024-04-26T20:12:31.017 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.pIGtR0tTzq is 6KB 2024-04-26T20:12:31.066 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.pIGtR0tTzq 2024-04-26T20:12:31.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T20:12:31.142 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T20:12:31.142 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T20:12:31.143 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T20:12:31.143 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T20:12:31.144 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T20:12:31.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-04-26T20:13:01.298 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-26T20:13:01.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-26T20:13:01.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T20:14:01.302 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T20:14:10.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-26T20:14:10.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T20:14:13.761 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-04-26T20:14:14.353 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-26T20:14:14.353 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T20:14:14.353 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-04-26T20:14:14.389 INFO:teuthology.orchestra.run.smithi102.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T20:14:14.389 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T20:14:14.389 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T20:14:14.389 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T20:14:15.392 DEBUG:teuthology.parallel:result is None 2024-04-26T20:14:15.392 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T20:14:15.400 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T20:14:15.400 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T20:14:15.405 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T20:14:15.409 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T20:14:15.448 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T20:14:15.455 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T20:14:15.519 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T20:14:15.520 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T20:14:15.525 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T20:14:15.527 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T20:14:15.871 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T20:14:15.877 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T20:14:15.878 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T20:14:15.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:14:15.894 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T20:14:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:14:15.912 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T20:14:15.918 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T20:14:15.938 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T20:14:15.979 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T20:14:15.996 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T20:14:16.021 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T20:14:16.025 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T20:14:16.034 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T20:14:16.069 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T20:14:16.073 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T20:14:16.080 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T20:14:16.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T20:14:16.084 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T20:14:16.088 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T20:14:16.144 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T20:14:16.154 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T20:14:16.154 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T20:14:16.159 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T20:14:16.162 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T20:14:16.196 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T20:14:16.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T20:14:16.405 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T20:14:16.475 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T20:14:16.560 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:14:16.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T20:14:16.647 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T20:14:16.693 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T20:14:16.752 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T20:14:16.796 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T20:14:16.836 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:14:16.836 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T20:14:16.902 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T20:14:16.947 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T20:14:17.009 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T20:14:17.052 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T20:14:17.094 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:14:17.094 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T20:14:17.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-04-26T20:14:17.165 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-04-26T20:14:17.167 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2024-04-26T20:14:17.200 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T20:14:17.246 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T20:14:17.256 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T20:14:17.611 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T20:14:17.619 INFO:teuthology.task.internal:Starting timer... 2024-04-26T20:14:17.619 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T20:14:17.628 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T20:14:17.636 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T20:14:17.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-04-26T20:14:17.742 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-04-26T20:14:17.747 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2024-04-26T20:14:17.816 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2024-04-26T20:14:17.818 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2024-04-26T20:14:17.883 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2024-04-26T20:14:17.885 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T20:14:17.885 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-04-26T20:14:17.932 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-04-26T20:14:17.933 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2024-04-26T20:14:17.959 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2024-04-26T20:14:17.959 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2024-04-26T20:14:17.992 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2024-04-26T20:14:17.992 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi112.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi130.front.sepia.ceph.com': 'permissive'} 2024-04-26T20:14:17.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T20:14:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:14:18.068 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T20:14:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:14:18.094 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T20:14:18.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:14:18.120 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T20:14:18.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-04-26T20:14:18.195 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2024-04-26T20:14:18.221 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2024-04-26T20:14:18.244 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T20:14:18.254 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-26T20:14:18.258 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-26T20:14:18.377 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T20:14:18.393 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T20:14:18.394 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi112.front.sepia.ceph.com,smithi130.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T20:19:17.080 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi130.front.sepia.ceph.com')] 2024-04-26T20:19:17.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-26T20:19:17.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T20:19:17.192 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-04-26T20:19:17.334 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-26T20:19:17.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-04-26T20:19:17.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T20:19:17.411 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-04-26T20:19:17.490 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-04-26T20:19:17.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-04-26T20:19:17.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T20:19:17.564 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-04-26T20:19:17.608 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-04-26T20:19:17.608 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T20:19:17.619 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T20:19:17.619 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T20:19:17.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T20:19:17.627 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T20:19:17.627 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T20:19:17.629 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T20:19:17.629 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T20:19:17.657 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T20:19:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T20:19:17.675 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T20:19:17.689 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T20:19:17.695 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2024-04-26T20:19:17.706 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2024-04-26T20:19:17.711 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2024-04-26T20:19:17.713 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T20:19:17.720 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T20:19:17.721 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2024-04-26T20:19:17.733 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T20:19:17.734 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T20:19:17.747 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T20:19:17.755 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T20:19:17.792 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-04-26T20:19:17.794 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T20:19:17.794 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-04-26T20:19:17.795 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:17.795 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:17.795 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:17.795 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:17.799 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-04-26T20:19:17.802 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T20:19:17.802 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-04-26T20:19:17.802 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:17.802 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:17.802 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:17.803 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:17.822 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-04-26T20:19:17.854 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-04-26T20:19:17.895 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T20:19:17.939 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T20:19:18.078 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-04-26T20:19:18.087 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T20:19:18.087 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-04-26T20:19:18.087 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:18.087 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:18.088 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:18.088 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:19:18.089 INFO:teuthology.run_tasks:Running task install... 2024-04-26T20:19:18.099 DEBUG:teuthology.task.install:project ceph 2024-04-26T20:19:18.099 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-26T20:19:18.099 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-26T20:19:18.099 INFO:teuthology.task.install:Using flavor: default 2024-04-26T20:19:18.105 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T20:19:18.105 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T20:19:18.106 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T20:19:18.106 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:19:18.109 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T20:19:18.109 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:19:18.110 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T20:19:18.110 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:19:18.255 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T20:19:18.255 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T20:19:18.332 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T20:19:18.332 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T20:19:18.336 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T20:19:18.336 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T20:19:18.439 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T20:19:18.439 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:19:18.439 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T20:19:18.458 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T20:19:18.458 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:19:18.458 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T20:19:18.467 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T20:19:18.467 DEBUG:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T20:19:18.483 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T20:19:18.483 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T20:19:18.524 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T20:19:18.524 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:19:18.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T20:19:18.535 DEBUG:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T20:19:18.548 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T20:19:18.611 DEBUG:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T20:19:18.614 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T20:19:18.614 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T20:19:18.620 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T20:19:18.676 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2024-04-26T20:19:18.678 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum clean all 2024-04-26T20:19:18.682 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2024-04-26T20:19:18.684 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2024-04-26T20:19:18.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T20:19:18.839 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T20:19:18.907 INFO:teuthology.orchestra.run.smithi130.stdout:48 files removed 2024-04-26T20:19:18.907 INFO:teuthology.orchestra.run.smithi112.stdout:48 files removed 2024-04-26T20:19:18.916 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-04-26T20:19:18.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-04-26T20:19:18.933 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T20:19:18.935 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T20:19:19.229 INFO:teuthology.orchestra.run.smithi102.stdout:48 files removed 2024-04-26T20:19:19.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T20:19:19.608 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 203 kB/s | 84 kB 00:00 2024-04-26T20:19:19.654 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 182 kB/s | 84 kB 00:00 2024-04-26T20:19:20.025 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-26T20:19:20.030 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-26T20:19:20.194 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 170 kB/s | 84 kB 00:00 2024-04-26T20:19:20.330 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 7.1 kB/s | 1.9 kB 00:00 2024-04-26T20:19:20.367 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-04-26T20:19:20.770 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-04-26T20:19:20.906 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-26T20:19:21.312 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 4.1 kB/s | 1.9 kB 00:00 2024-04-26T20:19:22.101 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-04-26T20:19:22.339 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 8.5 MB/s | 8.0 MB 00:00 2024-04-26T20:19:23.894 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-26T20:19:24.311 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-26T20:19:28.503 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 6.2 MB 00:01 2024-04-26T20:19:30.678 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 16 kB 00:01 2024-04-26T20:19:31.805 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-04-26T20:19:35.562 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 851 kB/s | 6.2 MB 00:07 2024-04-26T20:19:36.921 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-26T20:19:37.312 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-26T20:19:39.089 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T20:19:39.090 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T20:19:39.100 INFO:teuthology.orchestra.run.smithi130.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T20:19:39.101 INFO:teuthology.orchestra.run.smithi130.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T20:19:39.102 INFO:teuthology.orchestra.run.smithi130.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T20:19:39.103 INFO:teuthology.orchestra.run.smithi130.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T20:19:39.104 INFO:teuthology.orchestra.run.smithi130.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T20:19:39.106 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T20:19:39.107 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T20:19:39.168 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-04-26T20:19:39.182 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-04-26T20:19:39.182 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-04-26T20:19:39.182 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T20:19:39.183 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T20:19:39.184 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T20:19:39.185 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T20:19:39.186 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T20:19:39.187 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T20:19:39.188 INFO:teuthology.orchestra.run.smithi130.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T20:19:39.189 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T20:19:39.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T20:19:39.191 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T20:19:39.192 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T20:19:39.193 INFO:teuthology.orchestra.run.smithi130.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout:Installing weak dependencies: 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout:Install 148 Packages 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 4 Packages 2024-04-26T20:19:39.194 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:19:39.195 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 222 M 2024-04-26T20:19:39.195 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-04-26T20:19:40.206 INFO:teuthology.orchestra.run.smithi130.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-04-26T20:19:40.498 INFO:teuthology.orchestra.run.smithi130.stdout:(2/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 9.8 MB/s | 5.4 MB 00:00 2024-04-26T20:19:40.548 INFO:teuthology.orchestra.run.smithi130.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 2.9 MB/s | 145 kB 00:00 2024-04-26T20:19:40.707 INFO:teuthology.orchestra.run.smithi130.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 15 MB/s | 2.4 MB 00:00 2024-04-26T20:19:40.749 INFO:teuthology.orchestra.run.smithi130.stdout:(5/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.6 MB/s | 889 kB 00:00 2024-04-26T20:19:40.816 INFO:teuthology.orchestra.run.smithi130.stdout:(6/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 14 MB/s | 1.5 MB 00:00 2024-04-26T20:19:41.413 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 19 MB 00:19 2024-04-26T20:19:41.716 INFO:teuthology.orchestra.run.smithi130.stdout:(7/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x86 19 MB/s | 17 MB 00:00 2024-04-26T20:19:42.416 INFO:teuthology.orchestra.run.smithi130.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 15 MB/s | 11 MB 00:00 2024-04-26T20:19:42.458 INFO:teuthology.orchestra.run.smithi130.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 607 kB/s | 25 kB 00:00 2024-04-26T20:19:42.541 INFO:teuthology.orchestra.run.smithi130.stdout:(10/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x8 2.6 MB/s | 4.7 MB 00:01 2024-04-26T20:19:42.709 INFO:teuthology.orchestra.run.smithi130.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 1.3 MB/s | 225 kB 00:00 2024-04-26T20:19:42.809 INFO:teuthology.orchestra.run.smithi130.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 271 kB/s | 27 kB 00:00 2024-04-26T20:19:42.909 INFO:teuthology.orchestra.run.smithi130.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 317 kB/s | 32 kB 00:00 2024-04-26T20:19:43.218 INFO:teuthology.orchestra.run.smithi130.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 2.4 MB/s | 747 kB 00:00 2024-04-26T20:19:43.326 INFO:teuthology.orchestra.run.smithi130.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.5 MB/s | 163 kB 00:00 2024-04-26T20:19:43.427 INFO:teuthology.orchestra.run.smithi130.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.2 MB/s | 127 kB 00:00 2024-04-26T20:19:43.677 INFO:teuthology.orchestra.run.smithi130.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 2.0 MB/s | 501 kB 00:00 2024-04-26T20:19:45.328 INFO:teuthology.orchestra.run.smithi130.stdout:(18/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 17 MB/s | 49 MB 00:02 2024-04-26T20:19:45.536 INFO:teuthology.orchestra.run.smithi130.stdout:(19/152): ceph-common-19.0.0-3306.gbc612ffb.el9 3.8 MB/s | 21 MB 00:05 2024-04-26T20:19:45.561 INFO:teuthology.orchestra.run.smithi130.stdout:(20/152): python3-ceph-argparse-19.0.0-3306.gbc 193 kB/s | 45 kB 00:00 2024-04-26T20:19:45.637 INFO:teuthology.orchestra.run.smithi130.stdout:(21/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 2.7 MB/s | 5.4 MB 00:01 2024-04-26T20:19:45.662 INFO:teuthology.orchestra.run.smithi130.stdout:(22/152): python3-ceph-common-19.0.0-3306.gbc61 1.1 MB/s | 135 kB 00:00 2024-04-26T20:19:45.687 INFO:teuthology.orchestra.run.smithi130.stdout:(23/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.3 MB/s | 161 kB 00:00 2024-04-26T20:19:45.729 INFO:teuthology.orchestra.run.smithi130.stdout:(24/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 4.4 MB/s | 298 kB 00:00 2024-04-26T20:19:45.754 INFO:teuthology.orchestra.run.smithi130.stdout:(25/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.5 MB/s | 99 kB 00:00 2024-04-26T20:19:45.780 INFO:teuthology.orchestra.run.smithi130.stdout:(26/152): python3-rados-19.0.0-3306.gbc612ffb.e 2.2 MB/s | 321 kB 00:00 2024-04-26T20:19:45.805 INFO:teuthology.orchestra.run.smithi130.stdout:(27/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-26T20:19:45.864 INFO:teuthology.orchestra.run.smithi130.stdout:(28/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-26T20:19:45.889 INFO:teuthology.orchestra.run.smithi130.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3306.g 409 kB/s | 34 kB 00:00 2024-04-26T20:19:45.947 INFO:teuthology.orchestra.run.smithi130.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 16 MB/s | 3.1 MB 00:00 2024-04-26T20:19:45.973 INFO:teuthology.orchestra.run.smithi130.stdout:(31/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.3 MB/s | 142 kB 00:00 2024-04-26T20:19:46.157 INFO:teuthology.orchestra.run.smithi130.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 1.3 MB/s | 249 kB 00:00 2024-04-26T20:19:46.265 INFO:teuthology.orchestra.run.smithi130.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 456 kB/s | 49 kB 00:00 2024-04-26T20:19:46.399 INFO:teuthology.orchestra.run.smithi130.stdout:(34/152): ceph-mgr-diskprediction-local-19.0.0- 16 MB/s | 7.4 MB 00:00 2024-04-26T20:19:46.424 INFO:teuthology.orchestra.run.smithi130.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3306.gb 104 kB/s | 16 kB 00:00 2024-04-26T20:19:46.458 INFO:teuthology.orchestra.run.smithi130.stdout:(36/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 4.6 MB/s | 273 kB 00:00 2024-04-26T20:19:46.499 INFO:teuthology.orchestra.run.smithi130.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 439 kB/s | 18 kB 00:00 2024-04-26T20:19:46.558 INFO:teuthology.orchestra.run.smithi130.stdout:(38/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 5.3 MB/s | 3.5 MB 00:00 2024-04-26T20:19:46.592 INFO:teuthology.orchestra.run.smithi130.stdout:(39/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 4.4 MB/s | 762 kB 00:00 2024-04-26T20:19:46.651 INFO:teuthology.orchestra.run.smithi130.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 534 kB/s | 80 kB 00:00 2024-04-26T20:19:46.726 INFO:teuthology.orchestra.run.smithi130.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-26T20:19:46.751 INFO:teuthology.orchestra.run.smithi130.stdout:(42/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-26T20:19:46.801 INFO:teuthology.orchestra.run.smithi130.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-04-26T20:19:46.827 INFO:teuthology.orchestra.run.smithi130.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-26T20:19:46.852 INFO:teuthology.orchestra.run.smithi130.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-26T20:19:46.877 INFO:teuthology.orchestra.run.smithi130.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-26T20:19:46.902 INFO:teuthology.orchestra.run.smithi130.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-26T20:19:46.936 INFO:teuthology.orchestra.run.smithi130.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 5.8 MB/s | 1.2 MB 00:00 2024-04-26T20:19:46.970 INFO:teuthology.orchestra.run.smithi130.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 7.3 MB/s | 689 kB 00:00 2024-04-26T20:19:46.995 INFO:teuthology.orchestra.run.smithi130.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-26T20:19:47.012 INFO:teuthology.orchestra.run.smithi130.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 516 kB/s | 21 kB 00:00 2024-04-26T20:19:47.045 INFO:teuthology.orchestra.run.smithi130.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 6.5 MB/s | 943 kB 00:00 2024-04-26T20:19:47.056 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.2 MB 00:02 2024-04-26T20:19:47.062 INFO:teuthology.orchestra.run.smithi130.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 650 kB/s | 33 kB 00:00 2024-04-26T20:19:47.087 INFO:teuthology.orchestra.run.smithi130.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-04-26T20:19:47.113 INFO:teuthology.orchestra.run.smithi130.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 361 kB/s | 18 kB 00:00 2024-04-26T20:19:47.138 INFO:teuthology.orchestra.run.smithi130.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 220 kB/s | 11 kB 00:00 2024-04-26T20:19:47.164 INFO:teuthology.orchestra.run.smithi130.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-26T20:19:47.189 INFO:teuthology.orchestra.run.smithi130.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-04-26T20:19:47.223 INFO:teuthology.orchestra.run.smithi130.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-26T20:19:47.264 INFO:teuthology.orchestra.run.smithi130.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 582 kB 00:00 2024-04-26T20:19:47.382 INFO:teuthology.orchestra.run.smithi130.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 160 kB/s | 25 kB 00:00 2024-04-26T20:19:47.415 INFO:teuthology.orchestra.run.smithi130.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 315 kB/s | 47 kB 00:00 2024-04-26T20:19:47.440 INFO:teuthology.orchestra.run.smithi130.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-26T20:19:47.466 INFO:teuthology.orchestra.run.smithi130.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-26T20:19:47.491 INFO:teuthology.orchestra.run.smithi130.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 705 kB/s | 35 kB 00:00 2024-04-26T20:19:47.558 INFO:teuthology.orchestra.run.smithi130.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.8 MB/s | 3.0 MB 00:00 2024-04-26T20:19:47.667 INFO:teuthology.orchestra.run.smithi130.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-04-26T20:19:47.709 INFO:teuthology.orchestra.run.smithi130.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-26T20:19:47.767 INFO:teuthology.orchestra.run.smithi130.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-26T20:19:47.792 INFO:teuthology.orchestra.run.smithi130.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-26T20:19:47.826 INFO:teuthology.orchestra.run.smithi130.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 5.6 MB/s | 911 kB 00:00 2024-04-26T20:19:47.859 INFO:teuthology.orchestra.run.smithi130.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-26T20:19:47.885 INFO:teuthology.orchestra.run.smithi130.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T20:19:47.977 INFO:teuthology.orchestra.run.smithi130.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 28 MB/s | 5.8 MB 00:00 2024-04-26T20:19:48.002 INFO:teuthology.orchestra.run.smithi130.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-04-26T20:19:48.027 INFO:teuthology.orchestra.run.smithi130.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-04-26T20:19:48.053 INFO:teuthology.orchestra.run.smithi130.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 508 kB/s | 26 kB 00:00 2024-04-26T20:19:48.078 INFO:teuthology.orchestra.run.smithi130.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-04-26T20:19:48.103 INFO:teuthology.orchestra.run.smithi130.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-26T20:19:48.129 INFO:teuthology.orchestra.run.smithi130.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 692 kB/s | 35 kB 00:00 2024-04-26T20:19:48.221 INFO:teuthology.orchestra.run.smithi130.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-26T20:19:48.246 INFO:teuthology.orchestra.run.smithi130.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-26T20:19:48.271 INFO:teuthology.orchestra.run.smithi130.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-26T20:19:48.298 INFO:teuthology.orchestra.run.smithi130.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.4 MB/s | 77 kB 00:00 2024-04-26T20:19:48.324 INFO:teuthology.orchestra.run.smithi130.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 805 kB/s | 42 kB 00:00 2024-04-26T20:19:48.357 INFO:teuthology.orchestra.run.smithi130.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-26T20:19:48.382 INFO:teuthology.orchestra.run.smithi130.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-04-26T20:19:48.408 INFO:teuthology.orchestra.run.smithi130.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-26T20:19:48.433 INFO:teuthology.orchestra.run.smithi130.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-26T20:19:48.458 INFO:teuthology.orchestra.run.smithi130.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-26T20:19:48.526 INFO:teuthology.orchestra.run.smithi130.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-26T20:19:48.550 INFO:teuthology.orchestra.run.smithi130.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 453 kB/s | 42 kB 00:00 2024-04-26T20:19:48.576 INFO:teuthology.orchestra.run.smithi130.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 882 kB/s | 44 kB 00:00 2024-04-26T20:19:48.601 INFO:teuthology.orchestra.run.smithi130.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-26T20:19:48.606 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-26T20:19:48.626 INFO:teuthology.orchestra.run.smithi130.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 712 kB/s | 36 kB 00:00 2024-04-26T20:19:48.652 INFO:teuthology.orchestra.run.smithi130.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 941 kB/s | 47 kB 00:00 2024-04-26T20:19:48.677 INFO:teuthology.orchestra.run.smithi130.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-26T20:19:48.702 INFO:teuthology.orchestra.run.smithi130.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-26T20:19:48.728 INFO:teuthology.orchestra.run.smithi130.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-26T20:19:48.753 INFO:teuthology.orchestra.run.smithi130.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-26T20:19:48.770 INFO:teuthology.orchestra.run.smithi130.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.0 MB/s | 45 kB 00:00 2024-04-26T20:19:48.854 INFO:teuthology.orchestra.run.smithi130.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-04-26T20:19:49.071 INFO:teuthology.orchestra.run.smithi130.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-04-26T20:19:49.129 INFO:teuthology.orchestra.run.smithi130.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-04-26T20:19:49.238 INFO:teuthology.orchestra.run.smithi130.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-04-26T20:19:49.338 INFO:teuthology.orchestra.run.smithi130.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-26T20:19:49.388 INFO:teuthology.orchestra.run.smithi130.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-04-26T20:19:49.430 INFO:teuthology.orchestra.run.smithi130.stdout:(108/152): abseil-cpp-20211102.0-3.el9.x86_64.r 951 kB/s | 548 kB 00:00 2024-04-26T20:19:49.455 INFO:teuthology.orchestra.run.smithi130.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 723 kB/s | 49 kB 00:00 2024-04-26T20:19:49.497 INFO:teuthology.orchestra.run.smithi130.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-26T20:19:49.523 INFO:teuthology.orchestra.run.smithi130.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-26T20:19:49.615 INFO:teuthology.orchestra.run.smithi130.stdout:(112/152): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-26T20:19:49.640 INFO:teuthology.orchestra.run.smithi130.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-26T20:19:49.665 INFO:teuthology.orchestra.run.smithi130.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 861 kB/s | 43 kB 00:00 2024-04-26T20:19:49.691 INFO:teuthology.orchestra.run.smithi130.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-04-26T20:19:49.742 INFO:teuthology.orchestra.run.smithi130.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-04-26T20:19:49.750 INFO:teuthology.orchestra.run.smithi130.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-26T20:19:49.783 INFO:teuthology.orchestra.run.smithi130.stdout:(118/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.9 MB/s | 839 kB 00:00 2024-04-26T20:19:49.808 INFO:teuthology.orchestra.run.smithi130.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-04-26T20:19:49.834 INFO:teuthology.orchestra.run.smithi130.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-04-26T20:19:49.859 INFO:teuthology.orchestra.run.smithi130.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 466 kB/s | 35 kB 00:00 2024-04-26T20:19:49.874 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-26T20:19:49.884 INFO:teuthology.orchestra.run.smithi130.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-26T20:19:49.935 INFO:teuthology.orchestra.run.smithi130.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-04-26T20:19:49.960 INFO:teuthology.orchestra.run.smithi130.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-26T20:19:50.002 INFO:teuthology.orchestra.run.smithi130.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-04-26T20:19:50.027 INFO:teuthology.orchestra.run.smithi130.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 191 kB/s | 18 kB 00:00 2024-04-26T20:19:50.052 INFO:teuthology.orchestra.run.smithi130.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-04-26T20:19:50.078 INFO:teuthology.orchestra.run.smithi130.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-26T20:19:50.111 INFO:teuthology.orchestra.run.smithi130.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 337 kB/s | 20 kB 00:00 2024-04-26T20:19:50.137 INFO:teuthology.orchestra.run.smithi130.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 790 kB/s | 46 kB 00:00 2024-04-26T20:19:50.170 INFO:teuthology.orchestra.run.smithi130.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-26T20:19:50.195 INFO:teuthology.orchestra.run.smithi130.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 987 kB/s | 58 kB 00:00 2024-04-26T20:19:50.221 INFO:teuthology.orchestra.run.smithi130.stdout:(133/152): python3-more-itertools-8.12.0-2.el9. 722 kB/s | 79 kB 00:00 2024-04-26T20:19:50.246 INFO:teuthology.orchestra.run.smithi130.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-26T20:19:50.271 INFO:teuthology.orchestra.run.smithi130.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-04-26T20:19:50.297 INFO:teuthology.orchestra.run.smithi130.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-26T20:19:50.322 INFO:teuthology.orchestra.run.smithi130.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-26T20:19:50.347 INFO:teuthology.orchestra.run.smithi130.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-04-26T20:19:50.372 INFO:teuthology.orchestra.run.smithi130.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-26T20:19:50.397 INFO:teuthology.orchestra.run.smithi130.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-04-26T20:19:50.423 INFO:teuthology.orchestra.run.smithi130.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-04-26T20:19:50.448 INFO:teuthology.orchestra.run.smithi130.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-04-26T20:19:50.474 INFO:teuthology.orchestra.run.smithi130.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-26T20:19:50.490 INFO:teuthology.orchestra.run.smithi130.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-26T20:19:50.524 INFO:teuthology.orchestra.run.smithi130.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-04-26T20:19:50.549 INFO:teuthology.orchestra.run.smithi130.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-04-26T20:19:50.583 INFO:teuthology.orchestra.run.smithi130.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-26T20:19:50.625 INFO:teuthology.orchestra.run.smithi130.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-26T20:19:50.800 INFO:teuthology.orchestra.run.smithi130.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 7.7 MB/s | 1.3 MB 00:00 2024-04-26T20:19:50.884 INFO:teuthology.orchestra.run.smithi130.stdout:(150/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 10 MB/s | 3.1 MB 00:00 2024-04-26T20:19:50.934 INFO:teuthology.orchestra.run.smithi130.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 18 MB/s | 2.4 MB 00:00 2024-04-26T20:19:51.309 INFO:teuthology.orchestra.run.smithi130.stdout:(152/152): librados2-19.0.0-3306.gbc612ffb.el9. 4.5 MB/s | 3.4 MB 00:00 2024-04-26T20:19:51.312 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-04-26T20:19:51.312 INFO:teuthology.orchestra.run.smithi130.stdout:Total 18 MB/s | 222 MB 00:12 2024-04-26T20:19:52.427 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-04-26T20:19:52.572 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-04-26T20:19:52.572 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-04-26T20:19:54.324 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-04-26T20:19:54.325 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-04-26T20:19:54.950 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-26T20:19:57.051 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:19 2024-04-26T20:19:57.067 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T20:19:57.068 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T20:19:57.079 INFO:teuthology.orchestra.run.smithi112.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T20:19:57.079 INFO:teuthology.orchestra.run.smithi112.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T20:19:57.080 INFO:teuthology.orchestra.run.smithi112.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T20:19:57.081 INFO:teuthology.orchestra.run.smithi112.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T20:19:57.082 INFO:teuthology.orchestra.run.smithi112.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T20:19:57.085 INFO:teuthology.orchestra.run.smithi112.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T20:19:57.086 INFO:teuthology.orchestra.run.smithi112.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T20:19:57.145 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-04-26T20:19:57.169 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-04-26T20:19:57.169 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-04-26T20:19:57.169 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-04-26T20:19:57.170 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-04-26T20:19:57.170 INFO:teuthology.orchestra.run.smithi112.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T20:19:57.170 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T20:19:57.170 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T20:19:57.170 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T20:19:57.170 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T20:19:57.170 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T20:19:57.170 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T20:19:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T20:19:57.172 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-04-26T20:19:57.173 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T20:19:57.174 INFO:teuthology.orchestra.run.smithi112.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T20:19:57.174 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T20:19:57.174 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T20:19:57.174 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T20:19:57.174 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T20:19:57.174 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T20:19:57.174 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T20:19:57.174 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T20:19:57.174 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T20:19:57.175 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T20:19:57.176 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T20:19:57.177 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T20:19:57.177 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T20:19:57.177 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T20:19:57.177 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T20:19:57.177 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T20:19:57.177 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T20:19:57.177 INFO:teuthology.orchestra.run.smithi112.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T20:19:57.177 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T20:19:57.177 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T20:19:57.177 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T20:19:57.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T20:19:57.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T20:19:57.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T20:19:57.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T20:19:57.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T20:19:57.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T20:19:57.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T20:19:57.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T20:19:57.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T20:19:57.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T20:19:57.179 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T20:19:57.180 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T20:19:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T20:19:57.182 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout:Installing weak dependencies: 2024-04-26T20:19:57.183 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T20:19:57.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:19:57.184 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-04-26T20:19:57.184 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-04-26T20:19:57.184 INFO:teuthology.orchestra.run.smithi112.stdout:Install 148 Packages 2024-04-26T20:19:57.184 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 4 Packages 2024-04-26T20:19:57.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:19:57.184 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 222 M 2024-04-26T20:19:57.185 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-04-26T20:19:58.526 INFO:teuthology.orchestra.run.smithi112.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-04-26T20:19:58.851 INFO:teuthology.orchestra.run.smithi112.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 2.7 MB/s | 889 kB 00:00 2024-04-26T20:19:58.935 INFO:teuthology.orchestra.run.smithi112.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 1.7 MB/s | 145 kB 00:00 2024-04-26T20:19:59.510 INFO:teuthology.orchestra.run.smithi112.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 4.1 MB/s | 2.4 MB 00:00 2024-04-26T20:19:59.585 INFO:teuthology.orchestra.run.smithi112.stdout:(5/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 3.8 MB/s | 5.4 MB 00:01 2024-04-26T20:19:59.869 INFO:teuthology.orchestra.run.smithi112.stdout:(6/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 13 MB/s | 21 MB 00:01 2024-04-26T20:19:59.945 INFO:teuthology.orchestra.run.smithi112.stdout:(7/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 3.5 MB/s | 1.5 MB 00:00 2024-04-26T20:20:00.256 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-04-26T20:20:00.354 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T20:20:00.394 INFO:teuthology.orchestra.run.smithi112.stdout:(8/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 5.8 MB/s | 4.7 MB 00:00 2024-04-26T20:20:00.503 INFO:teuthology.orchestra.run.smithi112.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 233 kB/s | 25 kB 00:00 2024-04-26T20:20:00.561 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T20:20:00.623 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T20:20:00.665 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T20:20:00.690 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T20:20:00.725 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T20:20:00.803 INFO:teuthology.orchestra.run.smithi112.stdout:(10/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 18 MB/s | 17 MB 00:00 2024-04-26T20:20:00.862 INFO:teuthology.orchestra.run.smithi112.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 3.8 MB/s | 225 kB 00:00 2024-04-26T20:20:00.912 INFO:teuthology.orchestra.run.smithi112.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 543 kB/s | 27 kB 00:00 2024-04-26T20:20:00.971 INFO:teuthology.orchestra.run.smithi112.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 544 kB/s | 32 kB 00:00 2024-04-26T20:20:01.062 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T20:20:01.071 INFO:teuthology.orchestra.run.smithi112.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 7.3 MB/s | 747 kB 00:00 2024-04-26T20:20:01.130 INFO:teuthology.orchestra.run.smithi112.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 2.7 MB/s | 163 kB 00:00 2024-04-26T20:20:01.166 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T20:20:01.188 INFO:teuthology.orchestra.run.smithi112.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 2.1 MB/s | 127 kB 00:00 2024-04-26T20:20:01.272 INFO:teuthology.orchestra.run.smithi112.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 5.9 MB/s | 501 kB 00:00 2024-04-26T20:20:01.287 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T20:20:01.325 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T20:20:01.417 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T20:20:01.454 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T20:20:01.607 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T20:20:01.631 INFO:teuthology.orchestra.run.smithi112.stdout:(18/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 15 MB/s | 5.4 MB 00:00 2024-04-26T20:20:01.651 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T20:20:01.679 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T20:20:01.681 INFO:teuthology.orchestra.run.smithi112.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 905 kB/s | 45 kB 00:00 2024-04-26T20:20:01.731 INFO:teuthology.orchestra.run.smithi112.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 2.6 MB/s | 135 kB 00:00 2024-04-26T20:20:01.782 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T20:20:01.798 INFO:teuthology.orchestra.run.smithi112.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 2.4 MB/s | 161 kB 00:00 2024-04-26T20:20:01.805 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T20:20:01.865 INFO:teuthology.orchestra.run.smithi112.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 4.7 MB/s | 321 kB 00:00 2024-04-26T20:20:01.876 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T20:20:01.898 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T20:20:01.934 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T20:20:01.941 INFO:teuthology.orchestra.run.smithi112.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.9 MB/s | 298 kB 00:00 2024-04-26T20:20:01.999 INFO:teuthology.orchestra.run.smithi112.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.7 MB/s | 99 kB 00:00 2024-04-26T20:20:02.050 INFO:teuthology.orchestra.run.smithi112.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-26T20:20:02.098 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T20:20:02.131 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T20:20:02.158 INFO:teuthology.orchestra.run.smithi112.stdout:(26/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 4.8 MB/s | 11 MB 00:02 2024-04-26T20:20:02.238 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T20:20:02.259 INFO:teuthology.orchestra.run.smithi112.stdout:(27/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 15 MB/s | 3.1 MB 00:00 2024-04-26T20:20:02.284 INFO:teuthology.orchestra.run.smithi112.stdout:(28/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 1.3 MB/s | 171 kB 00:00 2024-04-26T20:20:02.301 INFO:teuthology.orchestra.run.smithi112.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3306.g 819 kB/s | 34 kB 00:00 2024-04-26T20:20:02.334 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T20:20:02.343 INFO:teuthology.orchestra.run.smithi112.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 2.4 MB/s | 142 kB 00:00 2024-04-26T20:20:02.372 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T20:20:02.416 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T20:20:02.489 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T20:20:02.523 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T20:20:02.567 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T20:20:02.599 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T20:20:02.601 INFO:teuthology.orchestra.run.smithi112.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 12 MB/s | 3.5 MB 00:00 2024-04-26T20:20:02.627 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T20:20:02.660 INFO:teuthology.orchestra.run.smithi112.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 4.2 MB/s | 249 kB 00:00 2024-04-26T20:20:02.697 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T20:20:02.719 INFO:teuthology.orchestra.run.smithi112.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 845 kB/s | 49 kB 00:00 2024-04-26T20:20:02.750 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T20:20:02.780 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T20:20:02.786 INFO:teuthology.orchestra.run.smithi112.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 247 kB/s | 16 kB 00:00 2024-04-26T20:20:02.824 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T20:20:02.853 INFO:teuthology.orchestra.run.smithi112.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 4.0 MB/s | 273 kB 00:00 2024-04-26T20:20:02.901 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T20:20:02.921 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-26T20:20:02.936 INFO:teuthology.orchestra.run.smithi112.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 9.0 MB/s | 762 kB 00:00 2024-04-26T20:20:02.943 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T20:20:02.987 INFO:teuthology.orchestra.run.smithi112.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 365 kB/s | 18 kB 00:00 2024-04-26T20:20:02.987 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T20:20:03.015 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T20:20:03.085 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T20:20:03.129 INFO:teuthology.orchestra.run.smithi112.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 568 kB/s | 80 kB 00:00 2024-04-26T20:20:03.130 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T20:20:03.205 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T20:20:03.329 INFO:teuthology.orchestra.run.smithi112.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-04-26T20:20:03.339 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T20:20:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T20:20:03.412 INFO:teuthology.orchestra.run.smithi112.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-26T20:20:03.505 INFO:teuthology.orchestra.run.smithi112.stdout:(41/152): ceph-mgr-diskprediction-local-19.0.0- 6.4 MB/s | 7.4 MB 00:01 2024-04-26T20:20:03.538 INFO:teuthology.orchestra.run.smithi112.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-04-26T20:20:03.730 INFO:teuthology.orchestra.run.smithi112.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-04-26T20:20:03.814 INFO:teuthology.orchestra.run.smithi112.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-26T20:20:03.881 INFO:teuthology.orchestra.run.smithi112.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-26T20:20:03.914 INFO:teuthology.orchestra.run.smithi112.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 3.0 MB/s | 1.2 MB 00:00 2024-04-26T20:20:03.931 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T20:20:03.965 INFO:teuthology.orchestra.run.smithi112.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-04-26T20:20:04.011 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T20:20:04.040 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T20:20:04.092 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T20:20:04.129 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T20:20:04.171 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T20:20:04.316 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T20:20:04.374 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T20:20:04.429 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T20:20:04.502 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T20:20:04.533 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T20:20:04.599 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T20:20:04.650 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T20:20:04.686 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T20:20:04.729 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T20:20:04.771 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T20:20:04.907 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T20:20:04.961 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T20:20:04.990 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T20:20:05.032 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T20:20:05.126 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T20:20:05.165 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T20:20:05.205 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T20:20:05.231 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T20:20:05.256 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T20:20:05.259 INFO:teuthology.orchestra.run.smithi112.stdout:(48/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 10 MB/s | 49 MB 00:04 2024-04-26T20:20:05.330 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T20:20:05.363 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T20:20:05.365 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T20:20:05.385 INFO:teuthology.orchestra.run.smithi102.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T20:20:05.386 INFO:teuthology.orchestra.run.smithi102.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T20:20:05.387 INFO:teuthology.orchestra.run.smithi102.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T20:20:05.389 INFO:teuthology.orchestra.run.smithi102.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T20:20:05.391 INFO:teuthology.orchestra.run.smithi102.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T20:20:05.395 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T20:20:05.396 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T20:20:05.415 INFO:teuthology.orchestra.run.smithi112.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 459 kB/s | 689 kB 00:01 2024-04-26T20:20:05.458 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T20:20:05.469 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-26T20:20:05.487 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-04-26T20:20:05.487 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-26T20:20:05.487 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-04-26T20:20:05.487 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-26T20:20:05.487 INFO:teuthology.orchestra.run.smithi102.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T20:20:05.487 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T20:20:05.487 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T20:20:05.487 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T20:20:05.487 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T20:20:05.488 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T20:20:05.489 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T20:20:05.490 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T20:20:05.491 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T20:20:05.492 INFO:teuthology.orchestra.run.smithi102.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T20:20:05.493 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T20:20:05.494 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T20:20:05.495 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T20:20:05.496 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T20:20:05.497 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout:Installing weak dependencies: 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout:Install 148 Packages 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 4 Packages 2024-04-26T20:20:05.498 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:20:05.499 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T20:20:05.499 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 222 M 2024-04-26T20:20:05.499 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-26T20:20:05.524 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T20:20:05.559 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T20:20:05.598 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T20:20:05.630 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T20:20:05.659 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T20:20:05.694 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T20:20:05.707 INFO:teuthology.orchestra.run.smithi112.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 541 kB/s | 943 kB 00:01 2024-04-26T20:20:05.869 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T20:20:05.911 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T20:20:05.932 INFO:teuthology.orchestra.run.smithi112.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 42 kB/s | 21 kB 00:00 2024-04-26T20:20:05.945 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T20:20:05.982 INFO:teuthology.orchestra.run.smithi112.stdout:(52/152): boost-program-options-1.75.0-8.el9.x8 148 kB/s | 107 kB 00:00 2024-04-26T20:20:06.034 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T20:20:06.076 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T20:20:06.107 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T20:20:06.154 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T20:20:06.252 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T20:20:06.300 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T20:20:06.340 INFO:teuthology.orchestra.run.smithi102.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-04-26T20:20:06.499 INFO:teuthology.orchestra.run.smithi112.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 61 kB/s | 33 kB 00:00 2024-04-26T20:20:06.548 INFO:teuthology.orchestra.run.smithi102.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 4.2 MB/s | 889 kB 00:00 2024-04-26T20:20:06.620 INFO:teuthology.orchestra.run.smithi102.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 2.0 MB/s | 145 kB 00:00 2024-04-26T20:20:06.708 INFO:teuthology.orchestra.run.smithi112.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 314 kB/s | 313 kB 00:00 2024-04-26T20:20:06.873 INFO:teuthology.orchestra.run.smithi102.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 9.4 MB/s | 2.4 MB 00:00 2024-04-26T20:20:06.983 INFO:teuthology.orchestra.run.smithi112.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 38 kB/s | 18 kB 00:00 2024-04-26T20:20:07.028 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T20:20:07.040 INFO:teuthology.orchestra.run.smithi102.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 9.1 MB/s | 1.5 MB 00:00 2024-04-26T20:20:07.081 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T20:20:07.114 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T20:20:07.192 INFO:teuthology.orchestra.run.smithi112.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 24 kB/s | 11 kB 00:00 2024-04-26T20:20:07.208 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T20:20:07.273 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T20:20:07.341 INFO:teuthology.orchestra.run.smithi102.stdout:(6/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 4.2 MB/s | 5.4 MB 00:01 2024-04-26T20:20:07.360 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T20:20:07.408 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T20:20:07.482 INFO:teuthology.orchestra.run.smithi102.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 11 MB/s | 4.7 MB 00:00 2024-04-26T20:20:07.586 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T20:20:07.614 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T20:20:07.692 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T20:20:07.742 INFO:teuthology.orchestra.run.smithi112.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 211 kB/s | 160 kB 00:00 2024-04-26T20:20:07.775 INFO:teuthology.orchestra.run.smithi112.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 81 kB/s | 45 kB 00:00 2024-04-26T20:20:08.347 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T20:20:08.477 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T20:20:08.508 INFO:teuthology.orchestra.run.smithi102.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 10 MB/s | 11 MB 00:01 2024-04-26T20:20:08.566 INFO:teuthology.orchestra.run.smithi102.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 441 kB/s | 25 kB 00:00 2024-04-26T20:20:08.918 INFO:teuthology.orchestra.run.smithi112.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 510 kB/s | 582 kB 00:01 2024-04-26T20:20:09.068 INFO:teuthology.orchestra.run.smithi112.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 500 kB/s | 662 kB 00:01 2024-04-26T20:20:09.134 INFO:teuthology.orchestra.run.smithi102.stdout:(10/152): ceph-common-19.0.0-3306.gbc612ffb.el9 6.9 MB/s | 21 MB 00:03 2024-04-26T20:20:09.242 INFO:teuthology.orchestra.run.smithi102.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.1 MB/s | 225 kB 00:00 2024-04-26T20:20:09.325 INFO:teuthology.orchestra.run.smithi102.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 327 kB/s | 27 kB 00:00 2024-04-26T20:20:09.460 INFO:teuthology.orchestra.run.smithi112.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 47 kB/s | 25 kB 00:00 2024-04-26T20:20:09.568 INFO:teuthology.orchestra.run.smithi102.stdout:(13/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 7.6 MB/s | 17 MB 00:02 2024-04-26T20:20:09.584 INFO:teuthology.orchestra.run.smithi102.stdout:(14/152): libcephfs-devel-19.0.0-3306.gbc612ffb 123 kB/s | 32 kB 00:00 2024-04-26T20:20:09.618 INFO:teuthology.orchestra.run.smithi112.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 86 kB/s | 47 kB 00:00 2024-04-26T20:20:09.668 INFO:teuthology.orchestra.run.smithi102.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.9 MB/s | 163 kB 00:00 2024-04-26T20:20:09.701 INFO:teuthology.orchestra.run.smithi102.stdout:(16/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 5.5 MB/s | 747 kB 00:00 2024-04-26T20:20:09.707 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T20:20:09.743 INFO:teuthology.orchestra.run.smithi102.stdout:(17/152): librados-devel-19.0.0-3306.gbc612ffb. 1.7 MB/s | 127 kB 00:00 2024-04-26T20:20:09.760 INFO:teuthology.orchestra.run.smithi112.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 810 kB/s | 3.0 MB 00:03 2024-04-26T20:20:09.778 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T20:20:09.794 INFO:teuthology.orchestra.run.smithi102.stdout:(18/152): libradosstriper1-19.0.0-3306.gbc612ff 5.4 MB/s | 501 kB 00:00 2024-04-26T20:20:09.806 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T20:20:09.839 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T20:20:09.902 INFO:teuthology.orchestra.run.smithi102.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 420 kB/s | 45 kB 00:00 2024-04-26T20:20:10.054 INFO:teuthology.orchestra.run.smithi102.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 899 kB/s | 135 kB 00:00 2024-04-26T20:20:10.142 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T20:20:10.144 INFO:teuthology.orchestra.run.smithi102.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.8 MB/s | 161 kB 00:00 2024-04-26T20:20:10.185 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T20:20:10.267 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T20:20:10.278 INFO:teuthology.orchestra.run.smithi102.stdout:(22/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 10 MB/s | 5.4 MB 00:00 2024-04-26T20:20:10.289 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T20:20:10.319 INFO:teuthology.orchestra.run.smithi112.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 63 kB/s | 35 kB 00:00 2024-04-26T20:20:10.320 INFO:teuthology.orchestra.run.smithi102.stdout:(23/152): python3-rados-19.0.0-3306.gbc612ffb.e 1.8 MB/s | 321 kB 00:00 2024-04-26T20:20:10.326 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T20:20:10.354 INFO:teuthology.orchestra.run.smithi102.stdout:(24/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.9 MB/s | 298 kB 00:00 2024-04-26T20:20:10.386 INFO:teuthology.orchestra.run.smithi112.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 246 kB/s | 188 kB 00:00 2024-04-26T20:20:10.396 INFO:teuthology.orchestra.run.smithi102.stdout:(25/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.3 MB/s | 99 kB 00:00 2024-04-26T20:20:10.411 INFO:teuthology.orchestra.run.smithi112.stdout:(66/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 307 kB/s | 292 kB 00:00 2024-04-26T20:20:10.429 INFO:teuthology.orchestra.run.smithi102.stdout:(26/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-26T20:20:10.513 INFO:teuthology.orchestra.run.smithi102.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-26T20:20:10.605 INFO:teuthology.orchestra.run.smithi102.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 377 kB/s | 34 kB 00:00 2024-04-26T20:20:10.689 INFO:teuthology.orchestra.run.smithi102.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.7 MB/s | 142 kB 00:00 2024-04-26T20:20:10.781 INFO:teuthology.orchestra.run.smithi102.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 8.0 MB/s | 3.1 MB 00:00 2024-04-26T20:20:10.804 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T20:20:10.892 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T20:20:10.976 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T20:20:11.007 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T20:20:11.131 INFO:teuthology.orchestra.run.smithi102.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 8.0 MB/s | 3.5 MB 00:00 2024-04-26T20:20:11.240 INFO:teuthology.orchestra.run.smithi102.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.3 MB/s | 249 kB 00:00 2024-04-26T20:20:11.315 INFO:teuthology.orchestra.run.smithi102.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 666 kB/s | 49 kB 00:00 2024-04-26T20:20:11.399 INFO:teuthology.orchestra.run.smithi102.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 199 kB/s | 16 kB 00:00 2024-04-26T20:20:11.507 INFO:teuthology.orchestra.run.smithi102.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.5 MB/s | 273 kB 00:00 2024-04-26T20:20:11.641 INFO:teuthology.orchestra.run.smithi102.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 8.6 MB/s | 7.4 MB 00:00 2024-04-26T20:20:11.699 INFO:teuthology.orchestra.run.smithi102.stdout:(37/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 3.9 MB/s | 762 kB 00:00 2024-04-26T20:20:11.725 INFO:teuthology.orchestra.run.smithi102.stdout:(38/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 221 kB/s | 18 kB 00:00 2024-04-26T20:20:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:(39/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 12 MB/s | 49 MB 00:03 2024-04-26T20:20:12.686 INFO:teuthology.orchestra.run.smithi112.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 715 kB/s | 1.6 MB 00:02 2024-04-26T20:20:12.850 INFO:teuthology.orchestra.run.smithi102.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 70 kB/s | 80 kB 00:01 2024-04-26T20:20:13.229 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T20:20:13.270 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T20:20:13.400 INFO:teuthology.orchestra.run.smithi102.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 231 kB/s | 195 kB 00:00 2024-04-26T20:20:13.709 INFO:teuthology.orchestra.run.smithi102.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 405 kB/s | 803 kB 00:01 2024-04-26T20:20:13.742 INFO:teuthology.orchestra.run.smithi102.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 284 kB/s | 253 kB 00:00 2024-04-26T20:20:13.816 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T20:20:13.857 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T20:20:13.950 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T20:20:13.950 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T20:20:13.950 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T20:20:13.950 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:13.983 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T20:20:14.220 INFO:teuthology.orchestra.run.smithi112.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 594 kB/s | 911 kB 00:01 2024-04-26T20:20:14.418 INFO:teuthology.orchestra.run.smithi102.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 150 kB/s | 106 kB 00:00 2024-04-26T20:20:14.526 INFO:teuthology.orchestra.run.smithi102.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 173 kB/s | 135 kB 00:00 2024-04-26T20:20:14.912 INFO:teuthology.orchestra.run.smithi112.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 193 kB/s | 133 kB 00:00 2024-04-26T20:20:15.260 INFO:teuthology.orchestra.run.smithi102.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 149 kB/s | 125 kB 00:00 2024-04-26T20:20:15.469 INFO:teuthology.orchestra.run.smithi102.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 228 kB/s | 215 kB 00:00 2024-04-26T20:20:15.602 INFO:teuthology.orchestra.run.smithi102.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 571 kB/s | 1.2 MB 00:02 2024-04-26T20:20:15.803 INFO:teuthology.orchestra.run.smithi112.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 836 kB/s | 4.4 MB 00:05 2024-04-26T20:20:16.327 INFO:teuthology.orchestra.run.smithi102.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 148 kB/s | 107 kB 00:00 2024-04-26T20:20:16.437 INFO:teuthology.orchestra.run.smithi112.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 793 kB/s | 4.7 MB 00:06 2024-04-26T20:20:16.728 INFO:teuthology.orchestra.run.smithi102.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 470 kB/s | 689 kB 00:01 2024-04-26T20:20:16.903 INFO:teuthology.orchestra.run.smithi102.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 37 kB/s | 21 kB 00:00 2024-04-26T20:20:17.062 INFO:teuthology.orchestra.run.smithi112.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 140 kB/s | 87 kB 00:00 2024-04-26T20:20:17.103 INFO:teuthology.orchestra.run.smithi102.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 603 kB/s | 943 kB 00:01 2024-04-26T20:20:17.471 INFO:teuthology.orchestra.run.smithi112.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 620 kB/s | 1.0 MB 00:01 2024-04-26T20:20:17.579 INFO:teuthology.orchestra.run.smithi102.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 48 kB/s | 33 kB 00:00 2024-04-26T20:20:17.738 INFO:teuthology.orchestra.run.smithi102.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 311 kB/s | 313 kB 00:01 2024-04-26T20:20:18.205 INFO:teuthology.orchestra.run.smithi102.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 29 kB/s | 18 kB 00:00 2024-04-26T20:20:18.271 INFO:teuthology.orchestra.run.smithi112.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 305 kB/s | 244 kB 00:00 2024-04-26T20:20:18.321 INFO:teuthology.orchestra.run.smithi102.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 19 kB/s | 11 kB 00:00 2024-04-26T20:20:18.955 INFO:teuthology.orchestra.run.smithi102.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-04-26T20:20:19.064 INFO:teuthology.orchestra.run.smithi102.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 187 kB/s | 160 kB 00:00 2024-04-26T20:20:19.113 INFO:teuthology.orchestra.run.smithi112.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 295 kB/s | 248 kB 00:00 2024-04-26T20:20:19.630 INFO:teuthology.orchestra.run.smithi112.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 50 kB/s | 26 kB 00:00 2024-04-26T20:20:20.097 INFO:teuthology.orchestra.run.smithi112.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 41 kB/s | 19 kB 00:00 2024-04-26T20:20:20.264 INFO:teuthology.orchestra.run.smithi102.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 507 kB/s | 662 kB 00:01 2024-04-26T20:20:20.297 INFO:teuthology.orchestra.run.smithi102.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 473 kB/s | 582 kB 00:01 2024-04-26T20:20:20.805 INFO:teuthology.orchestra.run.smithi112.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 243 kB/s | 172 kB 00:00 2024-04-26T20:20:20.972 INFO:teuthology.orchestra.run.smithi102.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 38 kB/s | 25 kB 00:00 2024-04-26T20:20:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:(62/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 763 kB/s | 3.0 MB 00:04 2024-04-26T20:20:21.148 INFO:teuthology.orchestra.run.smithi102.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 56 kB/s | 47 kB 00:00 2024-04-26T20:20:21.339 INFO:teuthology.orchestra.run.smithi112.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 65 kB/s | 35 kB 00:00 2024-04-26T20:20:21.857 INFO:teuthology.orchestra.run.smithi102.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 50 kB/s | 35 kB 00:00 2024-04-26T20:20:21.956 INFO:teuthology.orchestra.run.smithi112.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 840 kB/s | 5.8 MB 00:07 2024-04-26T20:20:21.973 INFO:teuthology.orchestra.run.smithi102.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 220 kB/s | 188 kB 00:00 2024-04-26T20:20:22.018 INFO:teuthology.orchestra.run.smithi102.stdout:(66/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 281 kB/s | 292 kB 00:01 2024-04-26T20:20:22.748 INFO:teuthology.orchestra.run.smithi112.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 317 kB/s | 250 kB 00:00 2024-04-26T20:20:23.623 INFO:teuthology.orchestra.run.smithi112.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 254 kB/s | 222 kB 00:00 2024-04-26T20:20:24.248 INFO:teuthology.orchestra.run.smithi112.stdout:(83/152): python3-babel-2.9.1-2.el9.noarch.rpm 850 kB/s | 6.0 MB 00:07 2024-04-26T20:20:24.324 INFO:teuthology.orchestra.run.smithi112.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 110 kB/s | 77 kB 00:00 2024-04-26T20:20:24.632 INFO:teuthology.orchestra.run.smithi102.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 622 kB/s | 1.6 MB 00:02 2024-04-26T20:20:24.832 INFO:teuthology.orchestra.run.smithi112.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 72 kB/s | 42 kB 00:00 2024-04-26T20:20:25.157 INFO:teuthology.orchestra.run.smithi112.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 323 kB/s | 269 kB 00:00 2024-04-26T20:20:25.404 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T20:20:25.404 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2024-04-26T20:20:25.405 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2024-04-26T20:20:25.405 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2024-04-26T20:20:25.405 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2024-04-26T20:20:25.405 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2024-04-26T20:20:25.405 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /root 2024-04-26T20:20:25.405 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2024-04-26T20:20:25.405 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:25.524 INFO:teuthology.orchestra.run.smithi112.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 230 kB/s | 159 kB 00:00 2024-04-26T20:20:25.580 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T20:20:25.599 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T20:20:25.599 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:20:25.599 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T20:20:25.599 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:26.027 INFO:teuthology.orchestra.run.smithi112.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 321 kB/s | 279 kB 00:00 2024-04-26T20:20:26.077 INFO:teuthology.orchestra.run.smithi112.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 92 kB/s | 51 kB 00:00 2024-04-26T20:20:26.157 INFO:teuthology.orchestra.run.smithi102.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 598 kB/s | 911 kB 00:01 2024-04-26T20:20:26.602 INFO:teuthology.orchestra.run.smithi112.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 93 kB/s | 54 kB 00:00 2024-04-26T20:20:26.891 INFO:teuthology.orchestra.run.smithi102.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 182 kB/s | 133 kB 00:00 2024-04-26T20:20:27.075 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T20:20:27.101 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T20:20:27.101 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:20:27.101 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T20:20:27.101 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T20:20:27.101 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T20:20:27.101 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:27.161 INFO:teuthology.orchestra.run.smithi112.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 75 kB/s | 42 kB 00:00 2024-04-26T20:20:27.318 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T20:20:27.340 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T20:20:27.341 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:20:27.341 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T20:20:27.341 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T20:20:27.341 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T20:20:27.341 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:27.641 INFO:teuthology.orchestra.run.smithi102.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 799 kB/s | 4.4 MB 00:05 2024-04-26T20:20:27.719 INFO:teuthology.orchestra.run.smithi112.stdout:(92/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 818 kB/s | 5.1 MB 00:06 2024-04-26T20:20:27.746 INFO:teuthology.orchestra.run.smithi112.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 76 kB/s | 44 kB 00:00 2024-04-26T20:20:27.785 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T20:20:27.807 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T20:20:27.808 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:20:27.808 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T20:20:27.808 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T20:20:27.808 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T20:20:27.808 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:27.923 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T20:20:27.980 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T20:20:28.036 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T20:20:28.221 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T20:20:28.241 INFO:teuthology.orchestra.run.smithi102.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 760 kB/s | 4.7 MB 00:06 2024-04-26T20:20:28.269 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T20:20:28.303 INFO:teuthology.orchestra.run.smithi112.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 64 kB/s | 36 kB 00:00 2024-04-26T20:20:28.347 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T20:20:28.436 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T20:20:28.562 INFO:teuthology.orchestra.run.smithi112.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 262 kB/s | 221 kB 00:00 2024-04-26T20:20:28.568 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T20:20:28.619 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T20:20:28.651 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T20:20:28.679 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T20:20:28.970 INFO:teuthology.orchestra.run.smithi112.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 71 kB/s | 47 kB 00:00 2024-04-26T20:20:29.042 INFO:teuthology.orchestra.run.smithi102.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 110 kB/s | 87 kB 00:00 2024-04-26T20:20:29.164 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T20:20:29.186 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T20:20:29.210 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T20:20:29.280 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T20:20:29.309 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T20:20:29.512 INFO:teuthology.orchestra.run.smithi112.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 330 kB/s | 305 kB 00:00 2024-04-26T20:20:29.542 INFO:teuthology.orchestra.run.smithi102.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 544 kB/s | 1.0 MB 00:01 2024-04-26T20:20:29.587 INFO:teuthology.orchestra.run.smithi112.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 94 kB/s | 58 kB 00:00 2024-04-26T20:20:30.188 INFO:teuthology.orchestra.run.smithi112.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 139 kB/s | 94 kB 00:00 2024-04-26T20:20:30.213 INFO:teuthology.orchestra.run.smithi112.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 102 kB/s | 64 kB 00:00 2024-04-26T20:20:30.354 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T20:20:30.380 INFO:teuthology.orchestra.run.smithi112.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-04-26T20:20:30.394 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T20:20:30.402 INFO:teuthology.orchestra.run.smithi102.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 285 kB/s | 244 kB 00:00 2024-04-26T20:20:30.522 INFO:teuthology.orchestra.run.smithi112.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 5.9 MB/s | 863 kB 00:00 2024-04-26T20:20:30.697 INFO:teuthology.orchestra.run.smithi112.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 3.1 MB/s | 548 kB 00:00 2024-04-26T20:20:30.739 INFO:teuthology.orchestra.run.smithi112.stdout:(104/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 81 kB/s | 45 kB 00:00 2024-04-26T20:20:30.789 INFO:teuthology.orchestra.run.smithi112.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 3.3 MB/s | 309 kB 00:00 2024-04-26T20:20:30.806 INFO:teuthology.orchestra.run.smithi112.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-04-26T20:20:30.848 INFO:teuthology.orchestra.run.smithi112.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 623 kB/s | 26 kB 00:00 2024-04-26T20:20:30.915 INFO:teuthology.orchestra.run.smithi112.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 731 kB/s | 49 kB 00:00 2024-04-26T20:20:30.965 INFO:teuthology.orchestra.run.smithi112.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-26T20:20:31.041 INFO:teuthology.orchestra.run.smithi112.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-26T20:20:31.274 INFO:teuthology.orchestra.run.smithi112.stdout:(111/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.5 MB/s | 839 kB 00:00 2024-04-26T20:20:31.309 INFO:teuthology.orchestra.run.smithi102.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 273 kB/s | 248 kB 00:00 2024-04-26T20:20:31.363 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T20:20:31.386 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T20:20:31.488 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T20:20:31.533 INFO:teuthology.orchestra.run.smithi112.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-04-26T20:20:31.642 INFO:teuthology.orchestra.run.smithi112.stdout:(113/152): libarrow-9.0.0-11.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-04-26T20:20:31.667 INFO:teuthology.orchestra.run.smithi112.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 325 kB/s | 43 kB 00:00 2024-04-26T20:20:31.674 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T20:20:31.705 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T20:20:31.708 INFO:teuthology.orchestra.run.smithi112.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 777 kB/s | 32 kB 00:00 2024-04-26T20:20:31.728 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T20:20:31.728 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:20:31.728 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T20:20:31.728 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T20:20:31.729 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T20:20:31.729 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:31.734 INFO:teuthology.orchestra.run.smithi112.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 225 kB/s | 15 kB 00:00 2024-04-26T20:20:31.773 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T20:20:31.784 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T20:20:31.801 INFO:teuthology.orchestra.run.smithi112.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-26T20:20:31.843 INFO:teuthology.orchestra.run.smithi112.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-04-26T20:20:31.874 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T20:20:31.892 INFO:teuthology.orchestra.run.smithi102.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 44 kB/s | 26 kB 00:00 2024-04-26T20:20:31.901 INFO:teuthology.orchestra.run.smithi112.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-04-26T20:20:31.921 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T20:20:31.927 INFO:teuthology.orchestra.run.smithi112.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-04-26T20:20:31.952 INFO:teuthology.orchestra.run.smithi112.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-26T20:20:31.994 INFO:teuthology.orchestra.run.smithi112.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-04-26T20:20:32.046 INFO:teuthology.orchestra.run.smithi112.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-04-26T20:20:32.086 INFO:teuthology.orchestra.run.smithi112.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 424 kB/s | 18 kB 00:00 2024-04-26T20:20:32.128 INFO:teuthology.orchestra.run.smithi112.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 558 kB/s | 23 kB 00:00 2024-04-26T20:20:32.170 INFO:teuthology.orchestra.run.smithi112.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 466 kB/s | 19 kB 00:00 2024-04-26T20:20:32.229 INFO:teuthology.orchestra.run.smithi112.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 338 kB/s | 20 kB 00:00 2024-04-26T20:20:32.421 INFO:teuthology.orchestra.run.smithi112.stdout:(128/152): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-04-26T20:20:32.463 INFO:teuthology.orchestra.run.smithi112.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 3.8 MB/s | 2.0 MB 00:00 2024-04-26T20:20:32.467 INFO:teuthology.orchestra.run.smithi102.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 33 kB/s | 19 kB 00:00 2024-04-26T20:20:32.488 INFO:teuthology.orchestra.run.smithi112.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 693 kB/s | 46 kB 00:00 2024-04-26T20:20:32.530 INFO:teuthology.orchestra.run.smithi112.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-26T20:20:32.555 INFO:teuthology.orchestra.run.smithi112.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 861 kB/s | 58 kB 00:00 2024-04-26T20:20:32.614 INFO:teuthology.orchestra.run.smithi112.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 282 kB/s | 16 kB 00:00 2024-04-26T20:20:32.639 INFO:teuthology.orchestra.run.smithi112.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-26T20:20:32.673 INFO:teuthology.orchestra.run.smithi112.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-26T20:20:32.698 INFO:teuthology.orchestra.run.smithi112.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 480 kB/s | 28 kB 00:00 2024-04-26T20:20:32.782 INFO:teuthology.orchestra.run.smithi112.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 284 kB/s | 31 kB 00:00 2024-04-26T20:20:32.845 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T20:20:32.848 INFO:teuthology.orchestra.run.smithi112.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-26T20:20:32.874 INFO:teuthology.orchestra.run.smithi112.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-04-26T20:20:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T20:20:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T20:20:32.898 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:20:32.898 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T20:20:32.898 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T20:20:32.898 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T20:20:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:32.907 INFO:teuthology.orchestra.run.smithi112.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 609 kB/s | 36 kB 00:00 2024-04-26T20:20:32.951 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T20:20:32.966 INFO:teuthology.orchestra.run.smithi112.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 555 kB/s | 51 kB 00:00 2024-04-26T20:20:32.973 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T20:20:32.973 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:20:32.973 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T20:20:32.973 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:33.008 INFO:teuthology.orchestra.run.smithi112.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-26T20:20:33.018 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T20:20:33.033 INFO:teuthology.orchestra.run.smithi112.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-26T20:20:33.040 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T20:20:33.040 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:20:33.040 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T20:20:33.041 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T20:20:33.041 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T20:20:33.041 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 133 kB/s | 20 kB 00:00 2024-04-26T20:20:33.242 INFO:teuthology.orchestra.run.smithi112.stdout:(145/152): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-26T20:20:33.268 INFO:teuthology.orchestra.run.smithi112.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 1.6 MB/s | 427 kB 00:00 2024-04-26T20:20:33.326 INFO:teuthology.orchestra.run.smithi102.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 201 kB/s | 172 kB 00:00 2024-04-26T20:20:33.352 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T20:20:33.375 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T20:20:33.375 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:20:33.375 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T20:20:33.375 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T20:20:33.375 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T20:20:33.375 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:33.560 INFO:teuthology.orchestra.run.smithi112.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-04-26T20:20:34.027 INFO:teuthology.orchestra.run.smithi112.stdout:(148/152): librados2-19.0.0-3306.gbc612ffb.el9. 4.5 MB/s | 3.4 MB 00:00 2024-04-26T20:20:34.035 INFO:teuthology.orchestra.run.smithi102.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 49 kB/s | 35 kB 00:00 2024-04-26T20:20:34.169 INFO:teuthology.orchestra.run.smithi112.stdout:(149/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 5.2 MB/s | 3.1 MB 00:00 2024-04-26T20:20:34.460 INFO:teuthology.orchestra.run.smithi112.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 3.1 MB/s | 1.3 MB 00:00 2024-04-26T20:20:34.485 INFO:teuthology.orchestra.run.smithi102.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 779 kB/s | 5.8 MB 00:07 2024-04-26T20:20:34.844 INFO:teuthology.orchestra.run.smithi112.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.5 MB/s | 2.4 MB 00:00 2024-04-26T20:20:35.352 INFO:teuthology.orchestra.run.smithi102.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 290 kB/s | 250 kB 00:00 2024-04-26T20:20:36.244 INFO:teuthology.orchestra.run.smithi102.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 249 kB/s | 222 kB 00:00 2024-04-26T20:20:36.728 INFO:teuthology.orchestra.run.smithi102.stdout:(83/152): python3-babel-2.9.1-2.el9.noarch.rpm 795 kB/s | 6.0 MB 00:07 2024-04-26T20:20:37.103 INFO:teuthology.orchestra.run.smithi102.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 90 kB/s | 77 kB 00:00 2024-04-26T20:20:37.395 INFO:teuthology.orchestra.run.smithi102.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 64 kB/s | 42 kB 00:00 2024-04-26T20:20:37.998 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T20:20:38.020 INFO:teuthology.orchestra.run.smithi102.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 294 kB/s | 269 kB 00:00 2024-04-26T20:20:38.039 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T20:20:38.081 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T20:20:38.112 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T20:20:38.137 INFO:teuthology.orchestra.run.smithi102.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 214 kB/s | 159 kB 00:00 2024-04-26T20:20:38.140 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T20:20:38.326 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T20:20:38.370 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T20:20:38.399 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T20:20:38.796 INFO:teuthology.orchestra.run.smithi102.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 77 kB/s | 51 kB 00:00 2024-04-26T20:20:38.912 INFO:teuthology.orchestra.run.smithi102.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 313 kB/s | 279 kB 00:00 2024-04-26T20:20:39.005 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T20:20:39.047 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T20:20:39.068 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T20:20:39.504 INFO:teuthology.orchestra.run.smithi102.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 76 kB/s | 54 kB 00:00 2024-04-26T20:20:39.889 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T20:20:39.969 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T20:20:40.001 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T20:20:40.092 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T20:20:40.130 INFO:teuthology.orchestra.run.smithi102.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 67 kB/s | 42 kB 00:00 2024-04-26T20:20:40.140 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T20:20:40.140 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T20:20:40.208 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T20:20:40.241 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T20:20:40.241 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T20:20:40.315 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T20:20:40.380 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T20:20:40.555 INFO:teuthology.orchestra.run.smithi102.stdout:(92/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 801 kB/s | 5.1 MB 00:06 2024-04-26T20:20:40.772 INFO:teuthology.orchestra.run.smithi102.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 69 kB/s | 44 kB 00:00 2024-04-26T20:20:41.439 INFO:teuthology.orchestra.run.smithi102.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 250 kB/s | 221 kB 00:00 2024-04-26T20:20:41.464 INFO:teuthology.orchestra.run.smithi102.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 58 kB/s | 36 kB 00:00 2024-04-26T20:20:42.048 INFO:teuthology.orchestra.run.smithi102.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 78 kB/s | 47 kB 00:00 2024-04-26T20:20:42.423 INFO:teuthology.orchestra.run.smithi102.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 318 kB/s | 305 kB 00:00 2024-04-26T20:20:42.723 INFO:teuthology.orchestra.run.smithi102.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 86 kB/s | 58 kB 00:00 2024-04-26T20:20:43.140 INFO:teuthology.orchestra.run.smithi102.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 131 kB/s | 94 kB 00:00 2024-04-26T20:20:43.440 INFO:teuthology.orchestra.run.smithi102.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 89 kB/s | 64 kB 00:00 2024-04-26T20:20:43.657 INFO:teuthology.orchestra.run.smithi102.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 104 kB/s | 22 kB 00:00 2024-04-26T20:20:43.791 INFO:teuthology.orchestra.run.smithi102.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 69 kB/s | 45 kB 00:00 2024-04-26T20:20:43.849 INFO:teuthology.orchestra.run.smithi102.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 4.4 MB/s | 863 kB 00:00 2024-04-26T20:20:44.274 INFO:teuthology.orchestra.run.smithi102.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-26T20:20:44.308 INFO:teuthology.orchestra.run.smithi102.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 792 kB/s | 309 kB 00:00 2024-04-26T20:20:44.350 INFO:teuthology.orchestra.run.smithi102.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 266 kB/s | 19 kB 00:00 2024-04-26T20:20:44.484 INFO:teuthology.orchestra.run.smithi102.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 203 kB/s | 26 kB 00:00 2024-04-26T20:20:44.584 INFO:teuthology.orchestra.run.smithi102.stdout:(108/152): libarrow-9.0.0-11.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-26T20:20:44.609 INFO:teuthology.orchestra.run.smithi102.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 402 kB/s | 49 kB 00:00 2024-04-26T20:20:44.651 INFO:teuthology.orchestra.run.smithi102.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-26T20:20:44.693 INFO:teuthology.orchestra.run.smithi102.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-04-26T20:20:44.744 INFO:teuthology.orchestra.run.smithi102.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.0 MB/s | 839 kB 00:00 2024-04-26T20:20:44.803 INFO:teuthology.orchestra.run.smithi102.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 5.4 MB/s | 548 kB 00:00 2024-04-26T20:20:44.828 INFO:teuthology.orchestra.run.smithi102.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 521 kB/s | 43 kB 00:00 2024-04-26T20:20:44.878 INFO:teuthology.orchestra.run.smithi102.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 435 kB/s | 32 kB 00:00 2024-04-26T20:20:44.885 INFO:teuthology.orchestra.run.smithi112.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 885 kB/s | 16 MB 00:18 2024-04-26T20:20:44.889 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-04-26T20:20:44.889 INFO:teuthology.orchestra.run.smithi112.stdout:Total 4.6 MB/s | 222 MB 00:47 2024-04-26T20:20:44.904 INFO:teuthology.orchestra.run.smithi102.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-04-26T20:20:44.962 INFO:teuthology.orchestra.run.smithi102.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-26T20:20:44.996 INFO:teuthology.orchestra.run.smithi102.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-04-26T20:20:45.063 INFO:teuthology.orchestra.run.smithi102.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.1 MB/s | 303 kB 00:00 2024-04-26T20:20:45.088 INFO:teuthology.orchestra.run.smithi102.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 387 kB/s | 35 kB 00:00 2024-04-26T20:20:45.164 INFO:teuthology.orchestra.run.smithi102.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-26T20:20:45.247 INFO:teuthology.orchestra.run.smithi102.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-04-26T20:20:45.280 INFO:teuthology.orchestra.run.smithi102.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-26T20:20:45.322 INFO:teuthology.orchestra.run.smithi102.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 147 kB/s | 11 kB 00:00 2024-04-26T20:20:45.348 INFO:teuthology.orchestra.run.smithi102.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 267 kB/s | 18 kB 00:00 2024-04-26T20:20:45.390 INFO:teuthology.orchestra.run.smithi102.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 352 kB/s | 23 kB 00:00 2024-04-26T20:20:45.415 INFO:teuthology.orchestra.run.smithi102.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 294 kB/s | 19 kB 00:00 2024-04-26T20:20:45.457 INFO:teuthology.orchestra.run.smithi102.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 299 kB/s | 20 kB 00:00 2024-04-26T20:20:45.541 INFO:teuthology.orchestra.run.smithi102.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 8.2 MB/s | 1.0 MB 00:00 2024-04-26T20:20:45.566 INFO:teuthology.orchestra.run.smithi102.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 428 kB/s | 46 kB 00:00 2024-04-26T20:20:45.625 INFO:teuthology.orchestra.run.smithi102.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 958 kB/s | 79 kB 00:00 2024-04-26T20:20:45.650 INFO:teuthology.orchestra.run.smithi102.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 696 kB/s | 58 kB 00:00 2024-04-26T20:20:45.717 INFO:teuthology.orchestra.run.smithi102.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-26T20:20:45.742 INFO:teuthology.orchestra.run.smithi102.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 180 kB/s | 16 kB 00:00 2024-04-26T20:20:45.809 INFO:teuthology.orchestra.run.smithi102.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 992 kB/s | 90 kB 00:00 2024-04-26T20:20:45.834 INFO:teuthology.orchestra.run.smithi102.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 309 kB/s | 28 kB 00:00 2024-04-26T20:20:45.885 INFO:teuthology.orchestra.run.smithi102.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 413 kB/s | 31 kB 00:00 2024-04-26T20:20:45.919 INFO:teuthology.orchestra.run.smithi102.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-26T20:20:45.945 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-04-26T20:20:45.960 INFO:teuthology.orchestra.run.smithi102.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 814 kB/s | 59 kB 00:00 2024-04-26T20:20:46.002 INFO:teuthology.orchestra.run.smithi102.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 431 kB/s | 36 kB 00:00 2024-04-26T20:20:46.053 INFO:teuthology.orchestra.run.smithi102.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 558 kB/s | 51 kB 00:00 2024-04-26T20:20:46.087 INFO:teuthology.orchestra.run.smithi102.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-26T20:20:46.087 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-04-26T20:20:46.087 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-04-26T20:20:46.128 INFO:teuthology.orchestra.run.smithi102.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-26T20:20:46.195 INFO:teuthology.orchestra.run.smithi102.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.9 MB/s | 427 kB 00:00 2024-04-26T20:20:46.221 INFO:teuthology.orchestra.run.smithi102.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 218 kB/s | 20 kB 00:00 2024-04-26T20:20:46.279 INFO:teuthology.orchestra.run.smithi102.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-04-26T20:20:46.346 INFO:teuthology.orchestra.run.smithi102.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-26T20:20:46.425 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T20:20:46.425 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T20:20:46.425 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T20:20:46.426 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T20:20:46.427 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T20:20:46.439 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T20:20:46.439 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T20:20:46.439 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T20:20:46.439 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T20:20:46.440 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T20:20:46.441 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T20:20:46.442 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T20:20:46.442 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T20:20:46.442 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T20:20:46.442 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T20:20:46.442 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T20:20:46.442 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T20:20:46.442 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T20:20:46.442 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T20:20:46.442 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T20:20:46.442 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T20:20:46.443 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T20:20:46.444 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T20:20:46.444 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T20:20:46.444 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T20:20:46.444 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T20:20:46.444 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T20:20:46.444 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T20:20:46.444 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T20:20:46.444 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T20:20:46.444 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T20:20:46.445 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T20:20:46.446 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T20:20:46.447 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T20:20:46.447 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T20:20:46.447 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T20:20:46.447 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T20:20:46.447 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T20:20:46.447 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T20:20:46.447 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T20:20:46.447 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T20:20:46.447 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T20:20:46.447 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T20:20:46.448 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T20:20:46.449 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T20:20:46.596 INFO:teuthology.orchestra.run.smithi102.stdout:(148/152): librados2-19.0.0-3306.gbc612ffb.el9. 11 MB/s | 3.4 MB 00:00 2024-04-26T20:20:47.080 INFO:teuthology.orchestra.run.smithi102.stdout:(149/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 4.3 MB/s | 3.1 MB 00:00 2024-04-26T20:20:47.924 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-04-26T20:20:47.924 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-04-26T20:20:48.557 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T20:20:48.557 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:48.557 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2024-04-26T20:20:48.557 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.557 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.557 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.558 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.559 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T20:20:48.560 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T20:20:48.561 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T20:20:48.562 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T20:20:48.563 INFO:teuthology.orchestra.run.smithi130.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T20:20:48.564 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T20:20:48.565 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T20:20:48.566 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T20:20:48.567 INFO:teuthology.orchestra.run.smithi130.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T20:20:48.568 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T20:20:48.569 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T20:20:48.569 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T20:20:48.569 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:20:48.569 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-04-26T20:20:48.847 INFO:teuthology.orchestra.run.smithi102.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 614 kB/s | 1.3 MB 00:02 2024-04-26T20:20:48.942 DEBUG:teuthology.parallel:result is None 2024-04-26T20:20:50.397 INFO:teuthology.orchestra.run.smithi102.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 739 kB/s | 2.4 MB 00:03 2024-04-26T20:20:53.884 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-04-26T20:20:53.963 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T20:20:54.001 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T20:20:54.063 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T20:20:54.097 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T20:20:54.123 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T20:20:54.165 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T20:20:54.492 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T20:20:54.599 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T20:20:54.704 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T20:20:54.751 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T20:20:54.840 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T20:20:54.871 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T20:20:55.016 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T20:20:55.059 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T20:20:55.087 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T20:20:55.190 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T20:20:55.222 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T20:20:55.284 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T20:20:55.306 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T20:20:55.327 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T20:20:55.689 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T20:20:55.722 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T20:20:55.829 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T20:20:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T20:20:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T20:20:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T20:20:56.063 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T20:20:56.106 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T20:20:56.149 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T20:20:56.181 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T20:20:56.210 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T20:20:56.279 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T20:20:56.333 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T20:20:56.371 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T20:20:56.415 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T20:20:56.501 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T20:20:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T20:20:56.577 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T20:20:56.606 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T20:20:56.676 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T20:20:56.720 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T20:20:56.794 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T20:20:56.937 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T20:20:56.998 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T20:20:57.515 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T20:20:57.609 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T20:20:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T20:20:57.682 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T20:20:57.719 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T20:20:57.761 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T20:20:57.919 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T20:20:57.972 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T20:20:58.026 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T20:20:58.038 INFO:teuthology.orchestra.run.smithi102.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 870 kB/s | 16 MB 00:19 2024-04-26T20:20:58.054 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-26T20:20:58.055 INFO:teuthology.orchestra.run.smithi102.stdout:Total 4.2 MB/s | 222 MB 00:52 2024-04-26T20:20:58.091 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T20:20:58.123 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T20:20:58.160 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T20:20:58.206 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T20:20:58.242 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T20:20:58.285 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T20:20:58.327 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T20:20:58.460 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T20:20:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T20:20:58.563 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T20:20:58.604 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T20:20:58.697 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T20:20:58.738 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T20:20:58.778 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T20:20:58.803 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T20:20:58.828 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T20:20:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T20:20:59.029 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T20:20:59.060 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T20:20:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T20:20:59.115 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T20:20:59.154 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T20:20:59.185 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T20:20:59.215 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T20:20:59.250 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T20:20:59.283 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T20:20:59.325 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T20:20:59.359 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T20:20:59.444 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T20:20:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T20:20:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T20:20:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T20:20:59.672 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T20:20:59.722 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T20:21:00.435 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T20:21:00.485 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T20:21:00.511 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T20:21:00.609 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T20:21:00.668 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T20:21:00.747 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T20:21:00.797 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T20:21:00.917 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-26T20:21:00.973 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T20:21:01.002 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T20:21:01.080 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T20:21:01.160 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-26T20:21:01.160 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-26T20:21:01.706 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T20:21:01.824 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T20:21:03.045 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T20:21:03.140 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T20:21:03.168 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T20:21:03.201 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T20:21:03.511 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T20:21:03.558 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T20:21:03.646 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T20:21:03.668 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T20:21:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T20:21:03.819 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-26T20:21:03.819 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-26T20:21:04.181 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T20:21:04.209 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T20:21:04.280 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T20:21:04.302 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T20:21:06.574 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T20:21:06.615 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T20:21:07.156 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T20:21:07.202 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T20:21:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T20:21:07.290 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T20:21:07.291 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T20:21:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:07.337 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T20:21:10.129 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-26T20:21:10.279 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T20:21:10.387 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T20:21:10.498 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T20:21:10.574 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T20:21:10.607 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T20:21:10.732 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T20:21:11.183 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T20:21:11.321 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T20:21:11.499 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T20:21:11.530 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T20:21:11.657 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T20:21:11.705 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T20:21:12.039 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T20:21:12.101 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T20:21:12.127 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T20:21:12.275 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T20:21:12.303 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T20:21:12.394 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T20:21:12.420 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T20:21:12.483 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T20:21:12.827 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T20:21:12.879 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T20:21:13.119 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T20:21:13.229 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T20:21:13.279 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T20:21:13.346 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T20:21:13.454 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T20:21:13.508 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T20:21:13.567 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T20:21:13.612 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T20:21:13.652 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T20:21:13.758 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T20:21:13.822 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T20:21:13.863 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T20:21:13.919 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T20:21:14.024 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T20:21:14.058 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T20:21:14.112 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T20:21:14.148 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T20:21:14.271 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T20:21:14.334 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T20:21:14.431 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T20:21:14.598 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T20:21:14.663 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T20:21:15.364 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T20:21:15.477 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T20:21:15.513 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T20:21:15.559 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T20:21:15.629 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T20:21:15.694 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T20:21:15.944 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T20:21:16.008 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T20:21:16.087 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T20:21:16.179 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T20:21:16.216 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T20:21:16.254 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T20:21:16.321 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T20:21:16.363 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T20:21:16.426 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T20:21:16.484 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T20:21:16.695 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T20:21:16.745 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T20:21:16.793 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T20:21:16.838 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T20:21:16.966 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T20:21:17.015 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T20:21:17.072 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T20:21:17.106 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T20:21:17.138 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T20:21:17.330 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T20:21:17.507 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T20:21:17.553 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T20:21:17.591 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T20:21:17.645 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T20:21:17.706 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T20:21:17.791 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T20:21:17.850 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T20:21:17.913 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T20:21:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T20:21:18.084 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T20:21:18.161 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T20:21:18.326 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T20:21:18.415 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T20:21:18.484 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T20:21:18.575 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T20:21:18.631 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T20:21:18.631 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2024-04-26T20:21:18.632 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2024-04-26T20:21:18.632 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2024-04-26T20:21:18.632 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2024-04-26T20:21:18.632 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2024-04-26T20:21:18.632 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /root 2024-04-26T20:21:18.632 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2024-04-26T20:21:18.632 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:18.749 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T20:21:18.808 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T20:21:18.822 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T20:21:18.822 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:18.822 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T20:21:18.822 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:18.838 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T20:21:19.810 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T20:21:19.897 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T20:21:19.946 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T20:21:20.094 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T20:21:20.182 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T20:21:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T20:21:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T20:21:20.303 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:20.303 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T20:21:20.303 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T20:21:20.303 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T20:21:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:20.303 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T20:21:20.364 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T20:21:20.520 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T20:21:20.542 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T20:21:20.542 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:20.542 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T20:21:20.542 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T20:21:20.542 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T20:21:20.542 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:20.638 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T20:21:20.686 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T20:21:20.818 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T20:21:20.979 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T20:21:21.001 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T20:21:21.001 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:21.001 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T20:21:21.001 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T20:21:21.001 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T20:21:21.001 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:21.100 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T20:21:21.148 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T20:21:21.203 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T20:21:21.386 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T20:21:21.438 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T20:21:21.514 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T20:21:21.602 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T20:21:21.679 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T20:21:21.732 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T20:21:21.771 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T20:21:21.803 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T20:21:21.831 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T20:21:21.833 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T20:21:22.292 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T20:21:22.321 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T20:21:22.353 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T20:21:22.415 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T20:21:22.444 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T20:21:23.457 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T20:21:23.473 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T20:21:23.504 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T20:21:23.552 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T20:21:23.585 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T20:21:23.618 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T20:21:23.986 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T20:21:24.031 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T20:21:24.144 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T20:21:24.179 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T20:21:24.233 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T20:21:24.461 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T20:21:24.488 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T20:21:24.588 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T20:21:24.740 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T20:21:24.773 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T20:21:24.774 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T20:21:24.796 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T20:21:24.796 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:24.796 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T20:21:24.796 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T20:21:24.796 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T20:21:24.796 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:24.809 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T20:21:24.841 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T20:21:24.852 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T20:21:24.901 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T20:21:24.928 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T20:21:24.940 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T20:21:24.989 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T20:21:25.907 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T20:21:25.944 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T20:21:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T20:21:25.966 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:25.966 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T20:21:25.966 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T20:21:25.966 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T20:21:25.966 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:26.011 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T20:21:26.033 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T20:21:26.033 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:26.033 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T20:21:26.033 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:26.078 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T20:21:26.100 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T20:21:26.100 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:26.100 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T20:21:26.101 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T20:21:26.101 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T20:21:26.101 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:26.404 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T20:21:26.426 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T20:21:26.426 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:26.426 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T20:21:26.426 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T20:21:26.426 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T20:21:26.426 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:27.602 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T20:21:27.641 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T20:21:28.294 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T20:21:28.353 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T20:21:28.518 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T20:21:28.518 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T20:21:28.518 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T20:21:28.518 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:28.553 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T20:21:30.991 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T20:21:31.033 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T20:21:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T20:21:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T20:21:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T20:21:31.312 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T20:21:31.354 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T20:21:31.384 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T20:21:31.995 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T20:21:32.040 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T20:21:32.059 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T20:21:32.880 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T20:21:32.962 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T20:21:33.002 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T20:21:33.083 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T20:21:33.132 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T20:21:33.133 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T20:21:33.217 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T20:21:33.250 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T20:21:33.250 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T20:21:33.339 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T20:21:33.409 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T20:21:39.229 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T20:21:39.229 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T20:21:39.229 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T20:21:39.230 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T20:21:39.230 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T20:21:39.230 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T20:21:39.230 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T20:21:39.231 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T20:21:39.232 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T20:21:39.233 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T20:21:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T20:21:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T20:21:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T20:21:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T20:21:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T20:21:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T20:21:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T20:21:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T20:21:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T20:21:39.234 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T20:21:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T20:21:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T20:21:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T20:21:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T20:21:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T20:21:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T20:21:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T20:21:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T20:21:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T20:21:39.235 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T20:21:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T20:21:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T20:21:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T20:21:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T20:21:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T20:21:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T20:21:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T20:21:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T20:21:39.236 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T20:21:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T20:21:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T20:21:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T20:21:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T20:21:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T20:21:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T20:21:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T20:21:39.237 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T20:21:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T20:21:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T20:21:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T20:21:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T20:21:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T20:21:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T20:21:39.238 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T20:21:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T20:21:39.240 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T20:21:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T20:21:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T20:21:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T20:21:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T20:21:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T20:21:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T20:21:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T20:21:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T20:21:39.241 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T20:21:39.242 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T20:21:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T20:21:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T20:21:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T20:21:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T20:21:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T20:21:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T20:21:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T20:21:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T20:21:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T20:21:39.243 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T20:21:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T20:21:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T20:21:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T20:21:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T20:21:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T20:21:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T20:21:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T20:21:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T20:21:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T20:21:39.244 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T20:21:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T20:21:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T20:21:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T20:21:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T20:21:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T20:21:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T20:21:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T20:21:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T20:21:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T20:21:39.245 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T20:21:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T20:21:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T20:21:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T20:21:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T20:21:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T20:21:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T20:21:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T20:21:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T20:21:39.246 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T20:21:41.328 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T20:21:41.328 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-04-26T20:21:41.328 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-04-26T20:21:41.328 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-04-26T20:21:41.328 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-04-26T20:21:41.328 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-04-26T20:21:41.328 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-04-26T20:21:41.329 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-04-26T20:21:41.329 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T20:21:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:41.349 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2024-04-26T20:21:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T20:21:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T20:21:41.349 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-04-26T20:21:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T20:21:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T20:21:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T20:21:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T20:21:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T20:21:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T20:21:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T20:21:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T20:21:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T20:21:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T20:21:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T20:21:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T20:21:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T20:21:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T20:21:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T20:21:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T20:21:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T20:21:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T20:21:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T20:21:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T20:21:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T20:21:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T20:21:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T20:21:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T20:21:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T20:21:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T20:21:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T20:21:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T20:21:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T20:21:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T20:21:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T20:21:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T20:21:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T20:21:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T20:21:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T20:21:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T20:21:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-26T20:21:41.363 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-04-26T20:21:41.560 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T20:21:41.610 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T20:21:41.610 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:41.610 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T20:21:41.610 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:41.718 DEBUG:teuthology.parallel:result is None 2024-04-26T20:21:43.221 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T20:21:43.297 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T20:21:43.297 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:43.297 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T20:21:43.297 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T20:21:43.297 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T20:21:43.297 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:43.546 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T20:21:43.612 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T20:21:43.612 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:43.612 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T20:21:43.612 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T20:21:43.612 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T20:21:43.612 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:44.113 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T20:21:44.183 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T20:21:44.183 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:44.183 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T20:21:44.183 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T20:21:44.183 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T20:21:44.183 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:44.322 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T20:21:44.382 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T20:21:44.446 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T20:21:44.672 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T20:21:44.726 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T20:21:44.812 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T20:21:44.920 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T20:21:45.085 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T20:21:45.137 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T20:21:45.197 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T20:21:45.241 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T20:21:45.860 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T20:21:45.912 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T20:21:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T20:21:46.077 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T20:21:46.113 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T20:21:47.364 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T20:21:47.398 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T20:21:48.493 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T20:21:48.532 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T20:21:48.678 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T20:21:48.853 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T20:21:48.884 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T20:21:48.953 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T20:21:48.953 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:48.953 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T20:21:48.953 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T20:21:48.953 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T20:21:48.953 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:49.010 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T20:21:49.047 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T20:21:49.159 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T20:21:49.229 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T20:21:50.230 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T20:21:50.271 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T20:21:50.339 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T20:21:50.339 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:50.339 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T20:21:50.339 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T20:21:50.339 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T20:21:50.339 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:50.388 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T20:21:50.454 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T20:21:50.454 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:50.454 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T20:21:50.455 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:50.553 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T20:21:50.629 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T20:21:50.629 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:50.630 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T20:21:50.630 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T20:21:50.630 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T20:21:50.630 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:51.056 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T20:21:51.128 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T20:21:51.128 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T20:21:51.128 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T20:21:51.129 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T20:21:51.129 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T20:21:51.129 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:21:56.188 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T20:21:56.234 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T20:21:56.313 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T20:21:56.357 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T20:21:56.405 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T20:21:56.720 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T20:21:56.768 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T20:21:56.796 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T20:21:57.688 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T20:21:57.744 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T20:21:57.806 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T20:21:59.154 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T20:21:59.268 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T20:21:59.350 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T20:21:59.515 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T20:21:59.574 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T20:21:59.574 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T20:21:59.700 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T20:21:59.739 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T20:21:59.739 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T20:21:59.880 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T20:22:00.019 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T20:22:00.300 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T20:22:00.300 INFO:teuthology.orchestra.run.smithi102.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T20:22:00.300 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:22:05.967 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T20:22:05.967 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T20:22:05.967 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T20:22:05.967 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T20:22:05.967 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T20:22:05.967 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T20:22:05.968 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T20:22:05.969 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T20:22:05.970 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T20:22:05.971 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T20:22:05.972 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T20:22:05.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T20:22:05.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T20:22:05.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T20:22:05.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T20:22:05.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T20:22:05.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T20:22:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T20:22:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:22:08.382 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-04-26T20:22:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T20:22:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:22:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:22:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:22:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:22:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:22:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:22:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:22:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:22:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T20:22:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T20:22:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T20:22:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T20:22:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T20:22:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T20:22:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T20:22:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T20:22:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T20:22:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T20:22:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T20:22:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T20:22:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T20:22:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T20:22:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T20:22:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T20:22:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T20:22:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T20:22:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T20:22:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T20:22:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T20:22:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T20:22:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T20:22:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T20:22:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T20:22:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T20:22:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T20:22:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T20:22:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T20:22:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T20:22:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T20:22:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T20:22:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T20:22:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T20:22:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T20:22:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T20:22:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T20:22:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T20:22:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T20:22:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T20:22:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T20:22:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T20:22:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T20:22:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T20:22:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T20:22:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T20:22:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T20:22:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T20:22:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T20:22:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T20:22:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T20:22:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T20:22:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T20:22:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:22:08.396 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-26T20:22:08.966 DEBUG:teuthology.parallel:result is None 2024-04-26T20:22:08.967 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:22:09.112 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T20:22:09.169 INFO:teuthology.orchestra.run.smithi102.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T20:22:09.170 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T20:22:09.170 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T20:22:09.172 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:22:09.310 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T20:22:09.336 INFO:teuthology.orchestra.run.smithi112.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T20:22:09.336 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T20:22:09.336 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T20:22:09.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:22:09.474 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T20:22:09.499 INFO:teuthology.orchestra.run.smithi130.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T20:22:09.500 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T20:22:09.500 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T20:22:09.502 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T20:22:09.502 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:22:09.502 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T20:22:09.587 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:22:09.587 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T20:22:09.618 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:22:09.618 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T20:22:09.649 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T20:22:09.650 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:22:09.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T20:22:09.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T20:22:09.830 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:22:09.830 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T20:22:09.858 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T20:22:09.926 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:22:09.926 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T20:22:09.955 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T20:22:10.023 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T20:22:10.024 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:22:10.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T20:22:10.091 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T20:22:10.201 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:22:10.201 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T20:22:10.225 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T20:22:10.288 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:22:10.289 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T20:22:10.312 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T20:22:10.375 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T20:22:10.375 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:22:10.375 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T20:22:10.443 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T20:22:10.556 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:22:10.556 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T20:22:10.579 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T20:22:10.644 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:22:10.644 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T20:22:10.668 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T20:22:10.732 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T20:22:10.831 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-26T20:22:10.831 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:22:10.832 INFO:tasks.cephadm:Cluster fsid is a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:22:10.832 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T20:22:10.832 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.102', 'mon.b': '172.21.15.112', 'mon.c': '172.21.15.130'} 2024-04-26T20:22:10.832 INFO:tasks.cephadm:First mon is mon.a on smithi102 2024-04-26T20:22:10.832 INFO:tasks.cephadm:First mgr is x 2024-04-26T20:22:10.832 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T20:22:10.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2024-04-26T20:22:10.910 DEBUG:teuthology.orchestra.run.smithi112:> sudo hostname $(hostname -s) 2024-04-26T20:22:10.935 DEBUG:teuthology.orchestra.run.smithi130:> sudo hostname $(hostname -s) 2024-04-26T20:22:10.958 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T20:22:10.958 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:22:11.100 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78918/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3306-gbc612ffb', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3306.gbc612ffb'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 13:02:42.256491', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240426.111959', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T20:22:11.237 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240426.111959, sha1 bc612ffb9c1d5bc27536bc917a705262a36d4387 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:22:11.239 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm 2024-04-26T20:22:11.239 DEBUG:teuthology.orchestra.run.smithi102:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T20:22:11.722 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 20:22 /home/ubuntu/cephtest/cephadm 2024-04-26T20:22:11.722 DEBUG:teuthology.orchestra.run.smithi112:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T20:22:12.159 INFO:teuthology.orchestra.run.smithi112.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 20:22 /home/ubuntu/cephtest/cephadm 2024-04-26T20:22:12.159 DEBUG:teuthology.orchestra.run.smithi130:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T20:22:12.590 INFO:teuthology.orchestra.run.smithi130.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 20:22 /home/ubuntu/cephtest/cephadm 2024-04-26T20:22:12.590 DEBUG:teuthology.orchestra.run.smithi102:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T20:22:12.636 DEBUG:teuthology.orchestra.run.smithi112:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T20:22:12.654 DEBUG:teuthology.orchestra.run.smithi130:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T20:22:12.707 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 on all hosts... 2024-04-26T20:22:12.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T20:22:12.712 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T20:22:12.714 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T20:22:14.203 INFO:teuthology.orchestra.run.smithi130.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T20:22:14.254 INFO:teuthology.orchestra.run.smithi112.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T20:22:14.915 INFO:teuthology.orchestra.run.smithi102.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T20:22:53.471 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:22:53.472 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T20:22:53.472 INFO:teuthology.orchestra.run.smithi102.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T20:22:53.472 INFO:teuthology.orchestra.run.smithi102.stdout: "repo_digests": [ 2024-04-26T20:22:53.472 INFO:teuthology.orchestra.run.smithi102.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T20:22:53.472 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-26T20:22:53.472 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:23:02.134 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-04-26T20:23:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T20:23:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T20:23:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: "repo_digests": [ 2024-04-26T20:23:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T20:23:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-04-26T20:23:02.135 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-04-26T20:23:07.904 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-26T20:23:07.905 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T20:23:07.905 INFO:teuthology.orchestra.run.smithi130.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T20:23:07.905 INFO:teuthology.orchestra.run.smithi130.stdout: "repo_digests": [ 2024-04-26T20:23:07.905 INFO:teuthology.orchestra.run.smithi130.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T20:23:07.905 INFO:teuthology.orchestra.run.smithi130.stdout: ] 2024-04-26T20:23:07.905 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-26T20:23:07.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2024-04-26T20:23:08.017 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph 2024-04-26T20:23:08.048 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph 2024-04-26T20:23:08.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2024-04-26T20:23:08.161 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /etc/ceph 2024-04-26T20:23:08.189 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 777 /etc/ceph 2024-04-26T20:23:08.213 INFO:tasks.cephadm:Writing seed config... 2024-04-26T20:23:08.214 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T20:23:08.214 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T20:23:08.214 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T20:23:08.214 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T20:23:08.214 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T20:23:08.214 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T20:23:08.215 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-26T20:23:08.216 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T20:23:08.217 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T20:23:08.217 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-26T20:23:08.217 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T20:23:08.217 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T20:23:08.217 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T20:23:08.217 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T20:23:08.217 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T20:23:08.217 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T20:23:08.217 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T20:23:08.217 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-26T20:23:08.218 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:23:08.218 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T20:23:08.265 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a94c830a-040a-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T20:23:08.266 DEBUG:teuthology.orchestra.run.smithi102:mon.a> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T20:23:08.313 DEBUG:teuthology.orchestra.run.smithi102:mgr.x> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T20:23:08.358 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T20:23:08.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 -v bootstrap --fsid a94c830a-040a-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.102 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:23:08.625 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-26T20:23:08.625 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387', '-v', 'bootstrap', '--fsid', 'a94c830a-040a-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.102', '--skip-admin-label'] 2024-04-26T20:23:08.626 INFO:teuthology.orchestra.run.smithi102.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T20:23:08.626 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying podman|docker is present... 2024-04-26T20:23:08.680 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T20:23:08.681 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying lvm2 is present... 2024-04-26T20:23:08.681 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying time synchronization is in place... 2024-04-26T20:23:08.696 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T20:23:08.696 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T20:23:08.711 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T20:23:08.711 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-26T20:23:08.736 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-04-26T20:23:08.758 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-04-26T20:23:08.758 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-04-26T20:23:08.759 INFO:teuthology.orchestra.run.smithi102.stdout:Repeating the final host check... 2024-04-26T20:23:08.817 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T20:23:08.817 INFO:teuthology.orchestra.run.smithi102.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T20:23:08.817 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl is present 2024-04-26T20:23:08.818 INFO:teuthology.orchestra.run.smithi102.stdout:lvcreate is present 2024-04-26T20:23:08.834 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T20:23:08.834 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T20:23:08.853 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T20:23:08.853 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-26T20:23:08.875 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-04-26T20:23:08.895 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-04-26T20:23:08.895 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-04-26T20:23:08.896 INFO:teuthology.orchestra.run.smithi102.stdout:Host looks OK 2024-04-26T20:23:08.896 INFO:teuthology.orchestra.run.smithi102.stdout:Cluster fsid: a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:08.897 INFO:teuthology.orchestra.run.smithi102.stdout:Acquiring lock 140033673608352 on /run/cephadm/a94c830a-040a-11ef-bc93-c7b262605968.lock 2024-04-26T20:23:08.897 INFO:teuthology.orchestra.run.smithi102.stdout:Lock 140033673608352 acquired on /run/cephadm/a94c830a-040a-11ef-bc93-c7b262605968.lock 2024-04-26T20:23:08.899 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 3300 ... 2024-04-26T20:23:08.900 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 6789 ... 2024-04-26T20:23:08.900 INFO:teuthology.orchestra.run.smithi102.stdout:Base mon IP(s) is [172.21.15.102:3300, 172.21.15.102:6789], mon addrv is [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-04-26T20:23:08.915 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.102 metric 100 2024-04-26T20:23:08.916 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.102 metric 100 2024-04-26T20:23:08.928 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T20:23:08.928 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-26T20:23:08.928 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1685sec hoplimit 64 pref medium 2024-04-26T20:23:08.938 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T20:23:08.939 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T20:23:08.939 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T20:23:08.939 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-26T20:23:08.940 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-04-26T20:23:08.940 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T20:23:08.942 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-04-26T20:23:08.942 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-04-26T20:23:08.942 INFO:teuthology.orchestra.run.smithi102.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T20:23:08.945 INFO:teuthology.orchestra.run.smithi102.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T20:23:08.947 INFO:teuthology.orchestra.run.smithi102.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T20:23:09.564 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T20:23:09.565 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T20:23:09.565 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T20:23:09.565 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:f340d36a00bce3feed0ad509aeafc9092c1c625bf553b8a613c380a79149b87e 2024-04-26T20:23:09.565 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-26T20:23:09.566 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying config sha256:55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T20:23:09.566 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T20:23:10.827 INFO:teuthology.orchestra.run.smithi102.stdout:ceph: stdout ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T20:23:10.828 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph version: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T20:23:10.828 INFO:teuthology.orchestra.run.smithi102.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T20:23:11.961 INFO:teuthology.orchestra.run.smithi102.stdout:stat: stdout 167 167 2024-04-26T20:23:11.961 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial keys... 2024-04-26T20:23:13.124 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQAwDSxmW3BNGRAAIQzKq9yPu7c8xXlHhjKcsA== 2024-04-26T20:23:14.291 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQAxDSxmwrxPJRAAVUnY/s9N/PG/twvGuW2Lsw== 2024-04-26T20:23:15.452 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQAyDSxmNdlOLhAAd/cOS5x/3dUmp7eK9MoApA== 2024-04-26T20:23:15.453 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial monmap... 2024-04-26T20:23:16.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T20:23:16.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T20:23:16.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:16.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T20:23:16.559 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool for a [v2:172.21.15.102:3300,v1:172.21.15.102:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T20:23:16.559 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = quincy 2024-04-26T20:23:16.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: set fsid to a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:16.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T20:23:16.559 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:23:16.560 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mon... 2024-04-26T20:23:17.693 INFO:teuthology.orchestra.run.smithi102.stdout:create mon.a on 2024-04-26T20:23:18.064 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T20:23:18.297 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T20:23:18.591 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a94c830a-040a-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-a94c830a-040a-11ef-bc93-c7b262605968.target. 2024-04-26T20:23:18.591 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a94c830a-040a-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-a94c830a-040a-11ef-bc93-c7b262605968.target. 2024-04-26T20:23:18.999 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.a 2024-04-26T20:23:18.999 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T20:23:19.298 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a94c830a-040a-11ef-bc93-c7b262605968.target.wants/ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-a94c830a-040a-11ef-bc93-c7b262605968@.service. 2024-04-26T20:23:20.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:19 smithi102 podman[27693]: 2024-04-26 20:23:19.72811704 +0000 UTC m=+0.157137358 container create 6d90a58358dea4678ea29685c2d4e18b9adcb1c7ad1cb63c9b7fed04b0db90a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T20:23:20.017 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:20 smithi102 podman[27693]: 2024-04-26 20:23:20.014963262 +0000 UTC m=+0.443983587 container init 6d90a58358dea4678ea29685c2d4e18b9adcb1c7ad1cb63c9b7fed04b0db90a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:23:20.134 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T20:23:20.134 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-04-26T20:23:20.148 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T20:23:20.149 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-26T20:23:20.149 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-04-26T20:23:20.149 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T20:23:20.149 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon to start... 2024-04-26T20:23:20.149 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon... 2024-04-26T20:23:20.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:20 smithi102 podman[27693]: 2024-04-26 20:23:20.025795852 +0000 UTC m=+0.454816180 container start 6d90a58358dea4678ea29685c2d4e18b9adcb1c7ad1cb63c9b7fed04b0db90a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:23:20.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:20 smithi102 ceph-mon[27712]: mkfs a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:20.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:20 smithi102 bash[27693]: 6d90a58358dea4678ea29685c2d4e18b9adcb1c7ad1cb63c9b7fed04b0db90a9 2024-04-26T20:23:20.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:20 smithi102 ceph-mon[27712]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T20:23:20.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:20 smithi102 systemd[1]: Started Ceph mon.a for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:23:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T20:23:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: monmap epoch 1 2024-04-26T20:23:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: last_changed 2024-04-26T20:23:15.907271+0000 2024-04-26T20:23:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: created 2024-04-26T20:23:15.907271+0000 2024-04-26T20:23:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: min_mon_release 19 (squid) 2024-04-26T20:23:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: election_strategy: 1 2024-04-26T20:23:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-26T20:23:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: fsmap 2024-04-26T20:23:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T20:23:21.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: mgrmap e1: no daemons active 2024-04-26T20:23:21.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:21 smithi102 ceph-mon[27712]: from='client.? 172.21.15.102:0/2659889977' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T20:23:21.437 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T20:23:21.437 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout id: a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:21.437 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T20:23:21.437 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout services: 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.748918s) 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout data: 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T20:23:21.438 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.756+0000 7f4b71600700 1 Processor -- start 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.756+0000 7f4b71600700 1 -- start start 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.757+0000 7f4b71600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c10bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.757+0000 7f4b71600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c07cea0 con 0x7f4b6c109810 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.758+0000 7f4b6aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c10bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.758+0000 7f4b6aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c10bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48646/0 (socket says 172.21.15.102:48646) 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.758+0000 7f4b6aa00700 1 -- 172.21.15.102:0/2829494633 learned_addr learned my addr 172.21.15.102:0/2829494633 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.759+0000 7f4b6aa00700 1 -- 172.21.15.102:0/2829494633 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c10c140 con 0x7f4b6c109810 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.759+0000 7f4b6aa00700 1 --2- 172.21.15.102:0/2829494633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c10bc00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0098a0 tx=0x7f4b5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd5aec30c33f7ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.759+0000 7f4b69600700 1 -- 172.21.15.102:0/2829494633 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b5c004160 con 0x7f4b6c109810 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.760+0000 7f4b69600700 1 -- 172.21.15.102:0/2829494633 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b5c004300 con 0x7f4b6c109810 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.760+0000 7f4b69600700 1 -- 172.21.15.102:0/2829494633 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b5c0036a0 con 0x7f4b6c109810 2024-04-26T20:23:21.439 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.760+0000 7f4b71600700 1 -- 172.21.15.102:0/2829494633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 msgr2=0x7f4b6c10bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.760+0000 7f4b71600700 1 --2- 172.21.15.102:0/2829494633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c10bc00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0098a0 tx=0x7f4b5c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.760+0000 7f4b71600700 1 -- 172.21.15.102:0/2829494633 shutdown_connections 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.760+0000 7f4b71600700 1 --2- 172.21.15.102:0/2829494633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c10bc00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.760+0000 7f4b71600700 1 -- 172.21.15.102:0/2829494633 >> 172.21.15.102:0/2829494633 conn(0x7f4b6c103240 msgr2=0x7f4b6c105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.760+0000 7f4b71600700 1 -- 172.21.15.102:0/2829494633 shutdown_connections 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.760+0000 7f4b71600700 1 -- 172.21.15.102:0/2829494633 wait complete. 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.762+0000 7f4b71600700 1 Processor -- start 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.762+0000 7f4b71600700 1 -- start start 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.762+0000 7f4b71600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.762+0000 7f4b71600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c10c610 con 0x7f4b6c109810 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.763+0000 7f4b6aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:21.440 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.763+0000 7f4b6aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48662/0 (socket says 172.21.15.102:48662) 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.763+0000 7f4b6aa00700 1 -- 172.21.15.102:0/2659889977 learned_addr learned my addr 172.21.15.102:0/2659889977 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.763+0000 7f4b6aa00700 1 -- 172.21.15.102:0/2659889977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c19ec20 con 0x7f4b6c109810 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.763+0000 7f4b6aa00700 1 --2- 172.21.15.102:0/2659889977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c19e6e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009fb0 tx=0x7f4b5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.764+0000 7f4b63400700 1 -- 172.21.15.102:0/2659889977 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b5c003de0 con 0x7f4b6c109810 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.764+0000 7f4b63400700 1 -- 172.21.15.102:0/2659889977 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b5c011630 con 0x7f4b6c109810 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.764+0000 7f4b63400700 1 -- 172.21.15.102:0/2659889977 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b5c011930 con 0x7f4b6c109810 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.764+0000 7f4b71600700 1 -- 172.21.15.102:0/2659889977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c19eeb0 con 0x7f4b6c109810 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.764+0000 7f4b71600700 1 -- 172.21.15.102:0/2659889977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c1a1ba0 con 0x7f4b6c109810 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.764+0000 7f4b63400700 1 -- 172.21.15.102:0/2659889977 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4b5c024480 con 0x7f4b6c109810 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.764+0000 7f4b63400700 1 -- 172.21.15.102:0/2659889977 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4b5c01a7e0 con 0x7f4b6c109810 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.765+0000 7f4b71600700 1 -- 172.21.15.102:0/2659889977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b6c064a80 con 0x7f4b6c109810 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.768+0000 7f4b63400700 1 -- 172.21.15.102:0/2659889977 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f4b5c01aa80 con 0x7f4b6c109810 2024-04-26T20:23:21.441 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.847+0000 7f4b71600700 1 -- 172.21.15.102:0/2659889977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f4b6c02e680 con 0x7f4b6c109810 2024-04-26T20:23:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.848+0000 7f4b63400700 1 -- 172.21.15.102:0/2659889977 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f4b5c01ac60 con 0x7f4b6c109810 2024-04-26T20:23:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.849+0000 7f4b71600700 1 -- 172.21.15.102:0/2659889977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 msgr2=0x7f4b6c19e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.849+0000 7f4b71600700 1 --2- 172.21.15.102:0/2659889977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c19e6e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009fb0 tx=0x7f4b5c004750 comp rx=0 tx=0).stop 2024-04-26T20:23:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.849+0000 7f4b71600700 1 -- 172.21.15.102:0/2659889977 shutdown_connections 2024-04-26T20:23:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.849+0000 7f4b71600700 1 --2- 172.21.15.102:0/2659889977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b6c109810 0x7f4b6c19e6e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.849+0000 7f4b71600700 1 -- 172.21.15.102:0/2659889977 >> 172.21.15.102:0/2659889977 conn(0x7f4b6c103240 msgr2=0x7f4b6c1041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.849+0000 7f4b71600700 1 -- 172.21.15.102:0/2659889977 shutdown_connections 2024-04-26T20:23:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:20.849+0000 7f4b71600700 1 -- 172.21.15.102:0/2659889977 wait complete. 2024-04-26T20:23:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:mon is available 2024-04-26T20:23:21.442 INFO:teuthology.orchestra.run.smithi102.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T20:23:22.372 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:22 smithi102 ceph-mon[27712]: from='client.? 172.21.15.102:0/2682266643' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T20:23:22.372 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:22 smithi102 ceph-mon[27712]: from='client.? 172.21.15.102:0/2682266643' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T20:23:22.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:22.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T20:23:22.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:22.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T20:23:22.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-04-26T20:23:22.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T20:23:22.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T20:23:22.877 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T20:23:22.877 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T20:23:22.877 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:22.878 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T20:23:22.878 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T20:23:22.878 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:22.878 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T20:23:22.878 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T20:23:22.878 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T20:23:22.879 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T20:23:22.879 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T20:23:22.879 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T20:23:22.879 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.018+0000 7fdbb5000700 1 Processor -- start 2024-04-26T20:23:22.879 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.018+0000 7fdbb5000700 1 -- start start 2024-04-26T20:23:22.879 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.019+0000 7fdbb5000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb010b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:22.884 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.019+0000 7fdbb5000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb010b7b0 con 0x7fdbb010adc0 2024-04-26T20:23:22.884 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.020+0000 7fdbaf400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb010b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:22.884 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.020+0000 7fdbaf400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb010b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48670/0 (socket says 172.21.15.102:48670) 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.020+0000 7fdbaf400700 1 -- 172.21.15.102:0/1778369902 learned_addr learned my addr 172.21.15.102:0/1778369902 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.020+0000 7fdbaf400700 1 -- 172.21.15.102:0/1778369902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb010bf70 con 0x7fdbb010adc0 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.021+0000 7fdbaf400700 1 --2- 172.21.15.102:0/1778369902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb010b1e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdba00098a0 tx=0x7fdba0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7253819b4777c6f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.021+0000 7fdbae000700 1 -- 172.21.15.102:0/1778369902 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdba0004160 con 0x7fdbb010adc0 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.021+0000 7fdbae000700 1 -- 172.21.15.102:0/1778369902 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba0004300 con 0x7fdbb010adc0 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.021+0000 7fdbae000700 1 -- 172.21.15.102:0/1778369902 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdba00036a0 con 0x7fdbb010adc0 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.021+0000 7fdbb5000700 1 -- 172.21.15.102:0/1778369902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 msgr2=0x7fdbb010b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.022+0000 7fdbb5000700 1 --2- 172.21.15.102:0/1778369902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb010b1e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdba00098a0 tx=0x7fdba0009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.022+0000 7fdbb5000700 1 -- 172.21.15.102:0/1778369902 shutdown_connections 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.022+0000 7fdbb5000700 1 --2- 172.21.15.102:0/1778369902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb010b1e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.022+0000 7fdbb5000700 1 -- 172.21.15.102:0/1778369902 >> 172.21.15.102:0/1778369902 conn(0x7fdbb007e1f0 msgr2=0x7fdbb007e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.022+0000 7fdbb5000700 1 -- 172.21.15.102:0/1778369902 shutdown_connections 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.022+0000 7fdbb5000700 1 -- 172.21.15.102:0/1778369902 wait complete. 2024-04-26T20:23:22.885 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.023+0000 7fdbb5000700 1 Processor -- start 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.024+0000 7fdbb5000700 1 -- start start 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.024+0000 7fdbb5000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb019e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.024+0000 7fdbb5000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb010c4a0 con 0x7fdbb010adc0 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.024+0000 7fdbaf400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb019e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.024+0000 7fdbaf400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb019e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48684/0 (socket says 172.21.15.102:48684) 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.024+0000 7fdbaf400700 1 -- 172.21.15.102:0/2682266643 learned_addr learned my addr 172.21.15.102:0/2682266643 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.025+0000 7fdbaf400700 1 -- 172.21.15.102:0/2682266643 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb019ea00 con 0x7fdbb010adc0 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.025+0000 7fdbaf400700 1 --2- 172.21.15.102:0/2682266643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb019e4c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdba0000c00 tx=0x7fdba0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.025+0000 7fdba7e00700 1 -- 172.21.15.102:0/2682266643 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdba0003db0 con 0x7fdbb010adc0 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.025+0000 7fdba7e00700 1 -- 172.21.15.102:0/2682266643 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba0011630 con 0x7fdbb010adc0 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.025+0000 7fdbb5000700 1 -- 172.21.15.102:0/2682266643 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb019ec90 con 0x7fdbb010adc0 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.025+0000 7fdba7e00700 1 -- 172.21.15.102:0/2682266643 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdba0011950 con 0x7fdbb010adc0 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.025+0000 7fdbb5000700 1 -- 172.21.15.102:0/2682266643 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb01a1980 con 0x7fdbb010adc0 2024-04-26T20:23:22.886 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.026+0000 7fdba7e00700 1 -- 172.21.15.102:0/2682266643 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdba0011af0 con 0x7fdbb010adc0 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.026+0000 7fdba7e00700 1 -- 172.21.15.102:0/2682266643 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdba001c450 con 0x7fdbb010adc0 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.027+0000 7fdbb5000700 1 -- 172.21.15.102:0/2682266643 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbb006a540 con 0x7fdbb010adc0 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.030+0000 7fdba7e00700 1 -- 172.21.15.102:0/2682266643 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fdba0032020 con 0x7fdbb010adc0 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.107+0000 7fdbb5000700 1 -- 172.21.15.102:0/2682266643 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fdbb002e610 con 0x7fdbb010adc0 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.113+0000 7fdba7e00700 1 -- 172.21.15.102:0/2682266643 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+467 (secure 0 0 0) 0x7fdba0004260 con 0x7fdbb010adc0 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.114+0000 7fdba7e00700 1 -- 172.21.15.102:0/2682266643 <== mon.0 v2:172.21.15.102:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdba001c650 con 0x7fdbb010adc0 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.115+0000 7fdbb5000700 1 -- 172.21.15.102:0/2682266643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 msgr2=0x7fdbb019e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.115+0000 7fdbb5000700 1 --2- 172.21.15.102:0/2682266643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb019e4c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdba0000c00 tx=0x7fdba0004750 comp rx=0 tx=0).stop 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.115+0000 7fdbb5000700 1 -- 172.21.15.102:0/2682266643 shutdown_connections 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.115+0000 7fdbb5000700 1 --2- 172.21.15.102:0/2682266643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb010adc0 0x7fdbb019e4c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.115+0000 7fdbb5000700 1 -- 172.21.15.102:0/2682266643 >> 172.21.15.102:0/2682266643 conn(0x7fdbb007e1f0 msgr2=0x7fdbb0195120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.116+0000 7fdbb5000700 1 -- 172.21.15.102:0/2682266643 shutdown_connections 2024-04-26T20:23:22.887 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:22.116+0000 7fdbb5000700 1 -- 172.21.15.102:0/2682266643 wait complete. 2024-04-26T20:23:22.888 INFO:teuthology.orchestra.run.smithi102.stdout:Generating new minimal ceph.conf... 2024-04-26T20:23:23.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:23 smithi102 ceph-mon[27712]: from='client.? 172.21.15.102:0/2261028563' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:23:24.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.574+0000 7fb180400700 1 Processor -- start 2024-04-26T20:23:24.306 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.574+0000 7fb180400700 1 -- start start 2024-04-26T20:23:24.306 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.575+0000 7fb180400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17810bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:24.307 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.575+0000 7fb180400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb17807cea0 con 0x7fb178109810 2024-04-26T20:23:24.307 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.575+0000 7fb17da00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17810bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:24.307 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.575+0000 7fb17da00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17810bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48696/0 (socket says 172.21.15.102:48696) 2024-04-26T20:23:24.307 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.575+0000 7fb17da00700 1 -- 172.21.15.102:0/702574348 learned_addr learned my addr 172.21.15.102:0/702574348 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:24.307 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.576+0000 7fb17da00700 1 -- 172.21.15.102:0/702574348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17810c140 con 0x7fb178109810 2024-04-26T20:23:24.307 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.576+0000 7fb17da00700 1 --2- 172.21.15.102:0/702574348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17810bc00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0098a0 tx=0x7fb16c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546c9bf3fb344dba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:24.307 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.576+0000 7fb177e00700 1 -- 172.21.15.102:0/702574348 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb16c004160 con 0x7fb178109810 2024-04-26T20:23:24.307 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.576+0000 7fb177e00700 1 -- 172.21.15.102:0/702574348 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb16c004300 con 0x7fb178109810 2024-04-26T20:23:24.307 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.576+0000 7fb177e00700 1 -- 172.21.15.102:0/702574348 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb16c011af0 con 0x7fb178109810 2024-04-26T20:23:24.307 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.577+0000 7fb180400700 1 -- 172.21.15.102:0/702574348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 msgr2=0x7fb17810bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.577+0000 7fb180400700 1 --2- 172.21.15.102:0/702574348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17810bc00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0098a0 tx=0x7fb16c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.577+0000 7fb180400700 1 -- 172.21.15.102:0/702574348 shutdown_connections 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.577+0000 7fb180400700 1 --2- 172.21.15.102:0/702574348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17810bc00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.577+0000 7fb180400700 1 -- 172.21.15.102:0/702574348 >> 172.21.15.102:0/702574348 conn(0x7fb178103240 msgr2=0x7fb178105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.577+0000 7fb180400700 1 -- 172.21.15.102:0/702574348 shutdown_connections 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.577+0000 7fb180400700 1 -- 172.21.15.102:0/702574348 wait complete. 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.579+0000 7fb180400700 1 Processor -- start 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.579+0000 7fb180400700 1 -- start start 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.580+0000 7fb180400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.580+0000 7fb180400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb17810d110 con 0x7fb178109810 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.580+0000 7fb17da00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.580+0000 7fb17da00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48698/0 (socket says 172.21.15.102:48698) 2024-04-26T20:23:24.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.580+0000 7fb17da00700 1 -- 172.21.15.102:0/2261028563 learned_addr learned my addr 172.21.15.102:0/2261028563 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:24.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.581+0000 7fb17da00700 1 -- 172.21.15.102:0/2261028563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17819d280 con 0x7fb178109810 2024-04-26T20:23:24.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.581+0000 7fb17da00700 1 --2- 172.21.15.102:0/2261028563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17819cd40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb16c011ac0 tx=0x7fb16c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:24.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.581+0000 7fb176000700 1 -- 172.21.15.102:0/2261028563 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb16c003970 con 0x7fb178109810 2024-04-26T20:23:24.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.581+0000 7fb176000700 1 -- 172.21.15.102:0/2261028563 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb16c003b10 con 0x7fb178109810 2024-04-26T20:23:24.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.581+0000 7fb176000700 1 -- 172.21.15.102:0/2261028563 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb16c024610 con 0x7fb178109810 2024-04-26T20:23:24.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.581+0000 7fb180400700 1 -- 172.21.15.102:0/2261028563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17819d510 con 0x7fb178109810 2024-04-26T20:23:24.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.581+0000 7fb180400700 1 -- 172.21.15.102:0/2261028563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1781a0200 con 0x7fb178109810 2024-04-26T20:23:24.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.582+0000 7fb176000700 1 -- 172.21.15.102:0/2261028563 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb16c024bd0 con 0x7fb178109810 2024-04-26T20:23:24.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.582+0000 7fb176000700 1 -- 172.21.15.102:0/2261028563 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb16c0193c0 con 0x7fb178109810 2024-04-26T20:23:24.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.583+0000 7fb180400700 1 -- 172.21.15.102:0/2261028563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb178064a80 con 0x7fb178109810 2024-04-26T20:23:24.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.585+0000 7fb176000700 1 -- 172.21.15.102:0/2261028563 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fb16c01a9e0 con 0x7fb178109810 2024-04-26T20:23:24.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.662+0000 7fb180400700 1 -- 172.21.15.102:0/2261028563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb1781a03d0 con 0x7fb178109810 2024-04-26T20:23:24.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.662+0000 7fb176000700 1 -- 172.21.15.102:0/2261028563 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7fb1781a03d0 con 0x7fb178109810 2024-04-26T20:23:24.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.663+0000 7fb180400700 1 -- 172.21.15.102:0/2261028563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 msgr2=0x7fb17819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:24.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.663+0000 7fb180400700 1 --2- 172.21.15.102:0/2261028563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17819cd40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb16c011ac0 tx=0x7fb16c011ba0 comp rx=0 tx=0).stop 2024-04-26T20:23:24.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.663+0000 7fb180400700 1 -- 172.21.15.102:0/2261028563 shutdown_connections 2024-04-26T20:23:24.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.663+0000 7fb180400700 1 --2- 172.21.15.102:0/2261028563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb178109810 0x7fb17819cd40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:24.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.663+0000 7fb180400700 1 -- 172.21.15.102:0/2261028563 >> 172.21.15.102:0/2261028563 conn(0x7fb178103240 msgr2=0x7fb178103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:24.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.664+0000 7fb180400700 1 -- 172.21.15.102:0/2261028563 shutdown_connections 2024-04-26T20:23:24.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:23.664+0000 7fb180400700 1 -- 172.21.15.102:0/2261028563 wait complete. 2024-04-26T20:23:24.311 INFO:teuthology.orchestra.run.smithi102.stdout:Restarting the monitor... 2024-04-26T20:23:24.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:24 smithi102 systemd[1]: Stopping Ceph mon.a for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:23:25.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a[27708]: 2024-04-26T20:23:24.803+0000 7f3b36600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:23:25.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a[27708]: 2024-04-26T20:23:24.803+0000 7f3b36600700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T20:23:25.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:24 smithi102 podman[27966]: 2024-04-26 20:23:24.928106191 +0000 UTC m=+0.263773387 container died 6d90a58358dea4678ea29685c2d4e18b9adcb1c7ad1cb63c9b7fed04b0db90a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T20:23:25.709 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:25 smithi102 podman[27966]: 2024-04-26 20:23:25.521666378 +0000 UTC m=+0.857333557 container cleanup 6d90a58358dea4678ea29685c2d4e18b9adcb1c7ad1cb63c9b7fed04b0db90a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T20:23:25.709 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:25 smithi102 bash[27966]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a 2024-04-26T20:23:26.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:25 smithi102 podman[27980]: 2024-04-26 20:23:25.708517307 +0000 UTC m=+0.758223118 container remove 6d90a58358dea4678ea29685c2d4e18b9adcb1c7ad1cb63c9b7fed04b0db90a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:23:26.603 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 systemd[1]: ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T20:23:26.603 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 systemd[1]: Stopped Ceph mon.a for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:23:26.603 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 systemd[1]: ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.a.service: Consumed 1.917s CPU time. 2024-04-26T20:23:26.603 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 systemd[1]: Starting Ceph mon.a for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:23:26.859 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 podman[28075]: 2024-04-26 20:23:26.509582327 +0000 UTC m=+0.035393377 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:23:26.859 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 podman[28075]: 2024-04-26 20:23:26.609262599 +0000 UTC m=+0.135073652 container create 747c74129f19a9156b2e006f7432e9b4ab52326e07e7bebb0b5eb6ec64806570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-26T20:23:26.859 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 podman[28075]: 2024-04-26 20:23:26.846116343 +0000 UTC m=+0.371927439 container init 747c74129f19a9156b2e006f7432e9b4ab52326e07e7bebb0b5eb6ec64806570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:23:26.859 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 podman[28075]: 2024-04-26 20:23:26.858082468 +0000 UTC m=+0.383893528 container start 747c74129f19a9156b2e006f7432e9b4ab52326e07e7bebb0b5eb6ec64806570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T20:23:26.955 INFO:teuthology.orchestra.run.smithi102.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: pidfile_write: ignore empty --pid-file 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: load: jerasure load: lrc 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: RocksDB version: 7.9.2 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Git sha 0 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: DB SUMMARY 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: DB Session ID: GR8X9HMHEAPN1WGWEFZR 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: CURRENT file: CURRENT 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T20:23:27.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104704 ; 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.error_if_exists: 0 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.create_if_missing: 0 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.paranoid_checks: 1 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.env: 0x558f2eb6ecc0 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.info_log: 0x558f49357dc0 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.statistics: (nil) 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.use_fsync: 0 2024-04-26T20:23:27.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_log_file_size: 0 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.allow_fallocate: 1 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.use_direct_reads: 0 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.db_log_dir: 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.wal_dir: 2024-04-26T20:23:27.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T20:23:27.126 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T20:23:27.126 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T20:23:27.126 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T20:23:27.126 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T20:23:27.126 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T20:23:27.126 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T20:23:27.126 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T20:23:27.126 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.write_buffer_manager: 0x558f493290e0 2024-04-26T20:23:27.126 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.unordered_write: 0 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.row_cache: None 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.wal_filter: None 2024-04-26T20:23:27.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.two_write_queues: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.wal_compression: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.atomic_flush: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.log_readahead_size: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T20:23:27.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T20:23:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T20:23:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_background_jobs: 2 2024-04-26T20:23:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_background_compactions: -1 2024-04-26T20:23:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_subcompactions: 1 2024-04-26T20:23:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T20:23:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T20:23:27.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_open_files: -1 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_background_flushes: -1 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Compression algorithms supported: 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T20:23:27.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: kZSTD supported: 0 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: kXpressCompression supported: 0 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: kZlibCompression supported: 1 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: kSnappyCompression supported: 1 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: kLZ4Compression supported: 1 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: kBZip2Compression supported: 0 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.merge_operator: 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_filter: None 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T20:23:27.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558f49357f80) 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_type: 0 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_index_type: 0 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_shortening: 1 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: checksum: 4 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: no_block_cache: 0 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache: 0x558f4935b1f0 2024-04-26T20:23:27.132 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_options: 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: capacity : 536870912 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_shard_bits : 4 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: strict_capacity_limit : 0 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_compressed: (nil) 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: persistent_cache: (nil) 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size: 4096 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size_deviation: 10 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_restart_interval: 16 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_block_restart_interval: 1 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: metadata_block_size: 4096 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: partition_filters: 0 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: use_delta_encoding: 1 2024-04-26T20:23:27.133 INFO:journalctl@ceph.mon.a.smithi102.stdout: filter_policy: bloomfilter 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout: whole_key_filtering: 1 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout: verify_compression: 0 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout: format_version: 5 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_index_compression: 1 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_align: 0 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout: max_auto_readahead_size: 262144 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout: prepopulate_block_cache: 0 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout: initial_auto_readahead_size: 8192 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compression: NoCompression 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T20:23:27.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T20:23:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T20:23:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.num_levels: 7 2024-04-26T20:23:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T20:23:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T20:23:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T20:23:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T20:23:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T20:23:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T20:23:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T20:23:27.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T20:23:27.136 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T20:23:27.136 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T20:23:27.136 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T20:23:27.136 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T20:23:27.136 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T20:23:27.136 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T20:23:27.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T20:23:27.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.inplace_update_support: 0 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T20:23:27.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.bloom_locality: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.max_successive_merges: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.ttl: 2592000 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.enable_blob_files: false 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.min_blob_size: 0 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T20:23:27.140 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b83e9979-9cba-4683-b67c-fda5838ac9a1 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714163006916186, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714163006918469, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99913, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97799, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14060, "raw_average_key_size": 49, "raw_value_size": 90139, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714163006, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b83e9979-9cba-4683-b67c-fda5838ac9a1", "db_session_id": "GR8X9HMHEAPN1WGWEFZR", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714163006918668, "job": 1, "event": "recovery_finished"} 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T20:23:27.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558f49478000 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: DB pointer 0x558f49464000 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** DB Stats ** 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-26T20:23:27.142 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: L0 2/0 99.43 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 60.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: Sum 2/0 99.43 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 60.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 60.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 60.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T20:23:27.143 INFO:journalctl@ceph.mon.a.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative compaction: 0.00 GB write, 13.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval compaction: 0.00 GB write, 13.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache BinnedLRUCache@0x558f4935b1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: starting mon.a rank 0 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: mon.a@-1(???) e1 preinit fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: mon.a@-1(???).mds e1 new map 2024-04-26T20:23:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: mon.a@-1(???).mds e1 print_map 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout: e1 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout: legacy client fscid: -1 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout: No filesystems configured 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 bash[28075]: 747c74129f19a9156b2e006f7432e9b4ab52326e07e7bebb0b5eb6ec64806570 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 systemd[1]: Started Ceph mon.a for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:23:27.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T20:23:27.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: monmap epoch 1 2024-04-26T20:23:27.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:27.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: last_changed 2024-04-26T20:23:15.907271+0000 2024-04-26T20:23:27.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: created 2024-04-26T20:23:15.907271+0000 2024-04-26T20:23:27.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: min_mon_release 19 (squid) 2024-04-26T20:23:27.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: election_strategy: 1 2024-04-26T20:23:27.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-26T20:23:27.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: fsmap 2024-04-26T20:23:27.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T20:23:27.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:26 smithi102 ceph-mon[28094]: mgrmap e1: no daemons active 2024-04-26T20:23:28.383 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.641+0000 7fe4e2600700 1 Processor -- start 2024-04-26T20:23:28.384 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.641+0000 7fe4e2600700 1 -- start start 2024-04-26T20:23:28.384 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.642+0000 7fe4e2600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc07d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:28.385 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.642+0000 7fe4e2600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc07d810 con 0x7fe4dc07edd0 2024-04-26T20:23:28.385 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.642+0000 7fe4db400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc07d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:28.385 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.642+0000 7fe4db400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc07d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40176/0 (socket says 172.21.15.102:40176) 2024-04-26T20:23:28.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.642+0000 7fe4db400700 1 -- 172.21.15.102:0/1717271842 learned_addr learned my addr 172.21.15.102:0/1717271842 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:28.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.643+0000 7fe4db400700 1 -- 172.21.15.102:0/1717271842 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4dc07d990 con 0x7fe4dc07edd0 2024-04-26T20:23:28.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.643+0000 7fe4db400700 1 --2- 172.21.15.102:0/1717271842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc07d2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0098a0 tx=0x7fe4cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152afb989d2c1af2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:28.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.643+0000 7fe4da000700 1 -- 172.21.15.102:0/1717271842 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4cc004160 con 0x7fe4dc07edd0 2024-04-26T20:23:28.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.644+0000 7fe4da000700 1 -- 172.21.15.102:0/1717271842 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe4cc004300 con 0x7fe4dc07edd0 2024-04-26T20:23:28.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.644+0000 7fe4da000700 1 -- 172.21.15.102:0/1717271842 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4cc011af0 con 0x7fe4dc07edd0 2024-04-26T20:23:28.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.644+0000 7fe4e2600700 1 -- 172.21.15.102:0/1717271842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 msgr2=0x7fe4dc07d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:28.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.644+0000 7fe4e2600700 1 --2- 172.21.15.102:0/1717271842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc07d2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0098a0 tx=0x7fe4cc009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:28.389 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.644+0000 7fe4e2600700 1 -- 172.21.15.102:0/1717271842 shutdown_connections 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.644+0000 7fe4e2600700 1 --2- 172.21.15.102:0/1717271842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc07d2d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.644+0000 7fe4e2600700 1 -- 172.21.15.102:0/1717271842 >> 172.21.15.102:0/1717271842 conn(0x7fe4dc1043d0 msgr2=0x7fe4dc1067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.644+0000 7fe4e2600700 1 -- 172.21.15.102:0/1717271842 shutdown_connections 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.645+0000 7fe4e2600700 1 -- 172.21.15.102:0/1717271842 wait complete. 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.645+0000 7fe4e2600700 1 Processor -- start 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.646+0000 7fe4e2600700 1 -- start start 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.646+0000 7fe4e2600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.646+0000 7fe4e2600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc10f300 con 0x7fe4dc07edd0 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.646+0000 7fe4db400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.646+0000 7fe4db400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc19ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40184/0 (socket says 172.21.15.102:40184) 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.646+0000 7fe4db400700 1 -- 172.21.15.102:0/2537533119 learned_addr learned my addr 172.21.15.102:0/2537533119 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.647+0000 7fe4db400700 1 -- 172.21.15.102:0/2537533119 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4dc19f440 con 0x7fe4dc07edd0 2024-04-26T20:23:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.647+0000 7fe4db400700 1 --2- 172.21.15.102:0/2537533119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc19ef00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009fb0 tx=0x7fe4cc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.647+0000 7fe4d3e00700 1 -- 172.21.15.102:0/2537533119 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4cc0036a0 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.647+0000 7fe4d3e00700 1 -- 172.21.15.102:0/2537533119 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe4cc01a430 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.647+0000 7fe4d3e00700 1 -- 172.21.15.102:0/2537533119 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4cc019550 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.647+0000 7fe4e2600700 1 -- 172.21.15.102:0/2537533119 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4dc19f6d0 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.648+0000 7fe4e2600700 1 -- 172.21.15.102:0/2537533119 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4dc1a23c0 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.648+0000 7fe4d3e00700 1 -- 172.21.15.102:0/2537533119 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe4cc019be0 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.648+0000 7fe4d3e00700 1 -- 172.21.15.102:0/2537533119 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe4cc024730 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.649+0000 7fe4e2600700 1 -- 172.21.15.102:0/2537533119 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4dc064a80 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.652+0000 7fe4d3e00700 1 -- 172.21.15.102:0/2537533119 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fe4cc043ec0 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.727+0000 7fe4e2600700 1 -- 172.21.15.102:0/2537533119 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fe4dc1a2590 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.730+0000 7fe4d3e00700 1 -- 172.21.15.102:0/2537533119 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fe4cc032030 con 0x7fe4dc07edd0 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.731+0000 7fe4e2600700 1 -- 172.21.15.102:0/2537533119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 msgr2=0x7fe4dc19ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:28.391 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.731+0000 7fe4e2600700 1 --2- 172.21.15.102:0/2537533119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc19ef00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009fb0 tx=0x7fe4cc011940 comp rx=0 tx=0).stop 2024-04-26T20:23:28.392 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.731+0000 7fe4e2600700 1 -- 172.21.15.102:0/2537533119 shutdown_connections 2024-04-26T20:23:28.392 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.731+0000 7fe4e2600700 1 --2- 172.21.15.102:0/2537533119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4dc07edd0 0x7fe4dc19ef00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:28.392 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.731+0000 7fe4e2600700 1 -- 172.21.15.102:0/2537533119 >> 172.21.15.102:0/2537533119 conn(0x7fe4dc1043d0 msgr2=0x7fe4dc104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:28.392 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.732+0000 7fe4e2600700 1 -- 172.21.15.102:0/2537533119 shutdown_connections 2024-04-26T20:23:28.392 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:27.732+0000 7fe4e2600700 1 -- 172.21.15.102:0/2537533119 wait complete. 2024-04-26T20:23:28.392 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T20:23:28.417 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:23:28.417 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mgr... 2024-04-26T20:23:28.419 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T20:23:28.420 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T20:23:28.857 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.x 2024-04-26T20:23:28.857 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T20:23:29.110 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a94c830a-040a-11ef-bc93-c7b262605968.target.wants/ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-a94c830a-040a-11ef-bc93-c7b262605968@.service. 2024-04-26T20:23:29.146 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:28 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2537533119' entity='client.admin' 2024-04-26T20:23:29.398 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:29 smithi102 systemd[1]: Starting Ceph mgr.x for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:23:29.742 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:29 smithi102 podman[28310]: 2024-04-26 20:23:29.398167552 +0000 UTC m=+0.036494182 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T20:23:29.742 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:29 smithi102 podman[28310]: 2024-04-26 20:23:29.511665798 +0000 UTC m=+0.149992446 container create 20c510044d08a23fb5c0733dee46abd56944768c0aa11840b6651674f8246687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-26T20:23:29.742 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:29 smithi102 podman[28310]: 2024-04-26 20:23:29.731529957 +0000 UTC m=+0.369856612 container init 20c510044d08a23fb5c0733dee46abd56944768c0aa11840b6651674f8246687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:23:29.743 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:29 smithi102 podman[28310]: 2024-04-26 20:23:29.740924008 +0000 UTC m=+0.379250663 container start 20c510044d08a23fb5c0733dee46abd56944768c0aa11840b6651674f8246687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-26T20:23:29.852 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T20:23:29.853 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-04-26T20:23:29.869 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T20:23:29.869 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-26T20:23:29.869 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-04-26T20:23:29.869 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T20:23:29.887 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T20:23:29.887 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-04-26T20:23:29.902 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T20:23:29.902 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-26T20:23:29.902 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-04-26T20:23:29.902 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T20:23:29.902 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr to start... 2024-04-26T20:23:29.902 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr... 2024-04-26T20:23:30.050 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:29 smithi102 bash[28310]: 20c510044d08a23fb5c0733dee46abd56944768c0aa11840b6651674f8246687 2024-04-26T20:23:30.050 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:29 smithi102 systemd[1]: Started Ceph mgr.x for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:23:30.050 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:29 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:29.971+0000 7efecd193200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T20:23:30.333 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:30.229+0000 7efecd193200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:23:30.625 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:30.366+0000 7efecd193200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T20:23:30.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:30 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3751318448' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T20:23:30.898 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:30.782+0000 7efecd193200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T20:23:31.209 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:30.898+0000 7efecd193200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:31.548 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T20:23:31.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T20:23:31.557 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T20:23:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T20:23:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T20:23:20.101995+0000", 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.691+0000 7f2d77200700 1 Processor -- start 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.692+0000 7f2d77200700 1 -- start start 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.692+0000 7f2d77200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7007d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.692+0000 7f2d77200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7007d8c0 con 0x7f2d7007ee90 2024-04-26T20:23:31.560 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.693+0000 7f2d6fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7007d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.693+0000 7f2d6fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7007d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40216/0 (socket says 172.21.15.102:40216) 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.693+0000 7f2d6fe00700 1 -- 172.21.15.102:0/1273424462 learned_addr learned my addr 172.21.15.102:0/1273424462 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.693+0000 7f2d6fe00700 1 -- 172.21.15.102:0/1273424462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7007da40 con 0x7f2d7007ee90 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.693+0000 7f2d6fe00700 1 --2- 172.21.15.102:0/1273424462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7007d2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2d600098a0 tx=0x7f2d60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9bedab83b3d3e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.694+0000 7f2d6ea00700 1 -- 172.21.15.102:0/1273424462 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d60004160 con 0x7f2d7007ee90 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.694+0000 7f2d6ea00700 1 -- 172.21.15.102:0/1273424462 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2d60004300 con 0x7f2d7007ee90 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.694+0000 7f2d6ea00700 1 -- 172.21.15.102:0/1273424462 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d60011af0 con 0x7f2d7007ee90 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.694+0000 7f2d77200700 1 -- 172.21.15.102:0/1273424462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 msgr2=0x7f2d7007d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.694+0000 7f2d77200700 1 --2- 172.21.15.102:0/1273424462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7007d2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2d600098a0 tx=0x7f2d60009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.694+0000 7f2d77200700 1 -- 172.21.15.102:0/1273424462 shutdown_connections 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.694+0000 7f2d77200700 1 --2- 172.21.15.102:0/1273424462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7007d2f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.694+0000 7f2d77200700 1 -- 172.21.15.102:0/1273424462 >> 172.21.15.102:0/1273424462 conn(0x7f2d70104370 msgr2=0x7f2d70106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.695+0000 7f2d77200700 1 -- 172.21.15.102:0/1273424462 shutdown_connections 2024-04-26T20:23:31.561 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.695+0000 7f2d77200700 1 -- 172.21.15.102:0/1273424462 wait complete. 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.696+0000 7f2d77200700 1 Processor -- start 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.696+0000 7f2d77200700 1 -- start start 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.696+0000 7f2d77200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.696+0000 7f2d77200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7010f320 con 0x7f2d7007ee90 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.696+0000 7f2d6fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.696+0000 7f2d6fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7019ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40222/0 (socket says 172.21.15.102:40222) 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.696+0000 7f2d6fe00700 1 -- 172.21.15.102:0/3751318448 learned_addr learned my addr 172.21.15.102:0/3751318448 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.697+0000 7f2d6fe00700 1 -- 172.21.15.102:0/3751318448 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7019f470 con 0x7f2d7007ee90 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.697+0000 7f2d6fe00700 1 --2- 172.21.15.102:0/3751318448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7019ef30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2d60000c00 tx=0x7f2d60011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.697+0000 7f2d6cc00700 1 -- 172.21.15.102:0/3751318448 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d60003b20 con 0x7f2d7007ee90 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.697+0000 7f2d77200700 1 -- 172.21.15.102:0/3751318448 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d7019f700 con 0x7f2d7007ee90 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.697+0000 7f2d6cc00700 1 -- 172.21.15.102:0/3751318448 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2d6001a8c0 con 0x7f2d7007ee90 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.697+0000 7f2d6cc00700 1 -- 172.21.15.102:0/3751318448 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d60024940 con 0x7f2d7007ee90 2024-04-26T20:23:31.562 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.697+0000 7f2d77200700 1 -- 172.21.15.102:0/3751318448 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d701a23f0 con 0x7f2d7007ee90 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.698+0000 7f2d6cc00700 1 -- 172.21.15.102:0/3751318448 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2d60022020 con 0x7f2d7007ee90 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.698+0000 7f2d6cc00700 1 -- 172.21.15.102:0/3751318448 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2d6002d9c0 con 0x7f2d7007ee90 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.698+0000 7f2d77200700 1 -- 172.21.15.102:0/3751318448 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d70064a80 con 0x7f2d7007ee90 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.701+0000 7f2d6cc00700 1 -- 172.21.15.102:0/3751318448 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f2d6002d3f0 con 0x7f2d7007ee90 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.780+0000 7f2d77200700 1 -- 172.21.15.102:0/3751318448 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2d701a29a0 con 0x7f2d7007ee90 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.781+0000 7f2d6cc00700 1 -- 172.21.15.102:0/3751318448 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2d6001aa70 con 0x7f2d7007ee90 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.782+0000 7f2d77200700 1 -- 172.21.15.102:0/3751318448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 msgr2=0x7f2d7019ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.782+0000 7f2d77200700 1 --2- 172.21.15.102:0/3751318448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7019ef30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2d60000c00 tx=0x7f2d60011b70 comp rx=0 tx=0).stop 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.782+0000 7f2d77200700 1 -- 172.21.15.102:0/3751318448 shutdown_connections 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.782+0000 7f2d77200700 1 --2- 172.21.15.102:0/3751318448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d7007ee90 0x7f2d7019ef30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.782+0000 7f2d77200700 1 -- 172.21.15.102:0/3751318448 >> 172.21.15.102:0/3751318448 conn(0x7f2d70104370 msgr2=0x7f2d70104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.782+0000 7f2d77200700 1 -- 172.21.15.102:0/3751318448 shutdown_connections 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:30.782+0000 7f2d77200700 1 -- 172.21.15.102:0/3751318448 wait complete. 2024-04-26T20:23:31.563 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (1/15)... 2024-04-26T20:23:32.898 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:32 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:32.473+0000 7efecd193200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T20:23:32.899 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:32 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:32.694+0000 7efecd193200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T20:23:33.297 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:32 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:32.905+0000 7efecd193200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T20:23:33.298 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:33.296+0000 7efecd193200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T20:23:33.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:33.416+0000 7efecd193200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T20:23:33.932 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:33.671+0000 7efecd193200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T20:23:33.932 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:33.931+0000 7efecd193200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:23:34.654 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:34 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3747378519' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T20:23:35.011 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:34 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:34.735+0000 7efecd193200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T20:23:35.023 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:35.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-26T20:23:35.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:23:35.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:35.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-26T20:23:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T20:23:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T20:23:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T20:23:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T20:23:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T20:23:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T20:23:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T20:23:35.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:35.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T20:23:35.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:35.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T20:23:35.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T20:23:35.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:35.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T20:23:35.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T20:23:35.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T20:23:35.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T20:23:35.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T20:23:20.101995+0000", 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T20:23:35.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-26T20:23:35.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.208+0000 7f6f0ec00700 1 Processor -- start 2024-04-26T20:23:35.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.208+0000 7f6f0ec00700 1 -- start start 2024-04-26T20:23:35.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.209+0000 7f6f0ec00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0807d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:35.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.209+0000 7f6f0ec00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0807d810 con 0x7f6f0807ee20 2024-04-26T20:23:35.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.209+0000 7f6f0d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0807d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:35.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.209+0000 7f6f0d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0807d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40224/0 (socket says 172.21.15.102:40224) 2024-04-26T20:23:35.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.209+0000 7f6f0d800700 1 -- 172.21.15.102:0/3325716726 learned_addr learned my addr 172.21.15.102:0/3325716726 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:35.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.210+0000 7f6f0d800700 1 -- 172.21.15.102:0/3325716726 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0807d990 con 0x7f6f0807ee20 2024-04-26T20:23:35.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.210+0000 7f6f0d800700 1 --2- 172.21.15.102:0/3325716726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0807d2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0098a0 tx=0x7f6efc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2851f1639664450 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:35.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.210+0000 7f6f07e00700 1 -- 172.21.15.102:0/3325716726 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6efc004160 con 0x7f6f0807ee20 2024-04-26T20:23:35.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.210+0000 7f6f07e00700 1 -- 172.21.15.102:0/3325716726 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6efc004300 con 0x7f6f0807ee20 2024-04-26T20:23:35.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.210+0000 7f6f07e00700 1 -- 172.21.15.102:0/3325716726 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6efc011af0 con 0x7f6f0807ee20 2024-04-26T20:23:35.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.211+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3325716726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 msgr2=0x7f6f0807d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:35.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.211+0000 7f6f0ec00700 1 --2- 172.21.15.102:0/3325716726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0807d2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0098a0 tx=0x7f6efc009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:35.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.211+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3325716726 shutdown_connections 2024-04-26T20:23:35.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.211+0000 7f6f0ec00700 1 --2- 172.21.15.102:0/3325716726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0807d2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:35.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.211+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3325716726 >> 172.21.15.102:0/3325716726 conn(0x7f6f081040f0 msgr2=0x7f6f08106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:35.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.211+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3325716726 shutdown_connections 2024-04-26T20:23:35.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.211+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3325716726 wait complete. 2024-04-26T20:23:35.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.212+0000 7f6f0ec00700 1 Processor -- start 2024-04-26T20:23:35.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.212+0000 7f6f0ec00700 1 -- start start 2024-04-26T20:23:35.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.213+0000 7f6f0ec00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0819ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:35.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.213+0000 7f6f0ec00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0810f100 con 0x7f6f0807ee20 2024-04-26T20:23:35.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.213+0000 7f6f0d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0819ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:35.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.213+0000 7f6f0d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0819ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40228/0 (socket says 172.21.15.102:40228) 2024-04-26T20:23:35.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.213+0000 7f6f0d800700 1 -- 172.21.15.102:0/3747378519 learned_addr learned my addr 172.21.15.102:0/3747378519 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.213+0000 7f6f0d800700 1 -- 172.21.15.102:0/3747378519 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0819f240 con 0x7f6f0807ee20 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.213+0000 7f6f0d800700 1 --2- 172.21.15.102:0/3747378519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0819ed00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6efc011ac0 tx=0x7f6efc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.214+0000 7f6f06000700 1 -- 172.21.15.102:0/3747378519 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6efc003970 con 0x7f6f0807ee20 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.214+0000 7f6f06000700 1 -- 172.21.15.102:0/3747378519 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6efc003b10 con 0x7f6f0807ee20 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.214+0000 7f6f06000700 1 -- 172.21.15.102:0/3747378519 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6efc024610 con 0x7f6f0807ee20 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.214+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3747378519 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0819f4d0 con 0x7f6f0807ee20 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.214+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3747378519 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f081a21b0 con 0x7f6f0807ee20 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.214+0000 7f6f06000700 1 -- 172.21.15.102:0/3747378519 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6efc024c20 con 0x7f6f0807ee20 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.214+0000 7f6f06000700 1 -- 172.21.15.102:0/3747378519 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6efc019700 con 0x7f6f0807ee20 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.215+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3747378519 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f0806a540 con 0x7f6f0807ee20 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.220+0000 7f6f06000700 1 -- 172.21.15.102:0/3747378519 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f6efc01a9e0 con 0x7f6f0807ee20 2024-04-26T20:23:35.044 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.299+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3747378519 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6f081a2380 con 0x7f6f0807ee20 2024-04-26T20:23:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.300+0000 7f6f06000700 1 -- 172.21.15.102:0/3747378519 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f6efc01a430 con 0x7f6f0807ee20 2024-04-26T20:23:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.301+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3747378519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 msgr2=0x7f6f0819ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.301+0000 7f6f0ec00700 1 --2- 172.21.15.102:0/3747378519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0819ed00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6efc011ac0 tx=0x7f6efc011ba0 comp rx=0 tx=0).stop 2024-04-26T20:23:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.302+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3747378519 shutdown_connections 2024-04-26T20:23:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.302+0000 7f6f0ec00700 1 --2- 172.21.15.102:0/3747378519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f0807ee20 0x7f6f0819ed00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.302+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3747378519 >> 172.21.15.102:0/3747378519 conn(0x7f6f081040f0 msgr2=0x7f6f08104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.302+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3747378519 shutdown_connections 2024-04-26T20:23:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:34.302+0000 7f6f0ec00700 1 -- 172.21.15.102:0/3747378519 wait complete. 2024-04-26T20:23:35.045 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (2/15)... 2024-04-26T20:23:35.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:35 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:35.288+0000 7efecd193200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T20:23:35.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:35 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:35.421+0000 7efecd193200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T20:23:36.148 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:35 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:35.682+0000 7efecd193200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T20:23:36.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:36 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:36.317+0000 7efecd193200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T20:23:36.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:36 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:36.438+0000 7efecd193200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T20:23:37.050 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:36 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:36.749+0000 7efecd193200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T20:23:37.050 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:36 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:36.924+0000 7efecd193200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T20:23:37.051 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:37 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:37.049+0000 7efecd193200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T20:23:37.364 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:37 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:37.211+0000 7efecd193200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T20:23:37.621 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:37 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:37.592+0000 7efecd193200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T20:23:37.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:37 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/250516800' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T20:23:38.483 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:38.380+0000 7efecd193200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T20:23:38.501 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:38.501 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-26T20:23:38.501 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:38.502 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:38.503 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T20:23:38.504 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T20:23:20.101995+0000", 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T20:23:38.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.682+0000 7f707f200700 1 Processor -- start 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.682+0000 7f707f200700 1 -- start start 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.683+0000 7f707f200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f70781080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.683+0000 7f707f200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7078108690 con 0x7f7078107ca0 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.683+0000 7f7077e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f70781080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.683+0000 7f7077e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f70781080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45160/0 (socket says 172.21.15.102:45160) 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.683+0000 7f7077e00700 1 -- 172.21.15.102:0/739856296 learned_addr learned my addr 172.21.15.102:0/739856296 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.684+0000 7f7077e00700 1 -- 172.21.15.102:0/739856296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7078108ea0 con 0x7f7078107ca0 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.684+0000 7f7077e00700 1 --2- 172.21.15.102:0/739856296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f70781080c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f70680098a0 tx=0x7f7068009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97c6d30c9db7536 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.684+0000 7f7076a00700 1 -- 172.21.15.102:0/739856296 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7068004160 con 0x7f7078107ca0 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.684+0000 7f7076a00700 1 -- 172.21.15.102:0/739856296 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7068004300 con 0x7f7078107ca0 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.684+0000 7f7076a00700 1 -- 172.21.15.102:0/739856296 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7068011af0 con 0x7f7078107ca0 2024-04-26T20:23:38.506 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.685+0000 7f707f200700 1 -- 172.21.15.102:0/739856296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 msgr2=0x7f70781080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.685+0000 7f707f200700 1 --2- 172.21.15.102:0/739856296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f70781080c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f70680098a0 tx=0x7f7068009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.686+0000 7f707f200700 1 -- 172.21.15.102:0/739856296 shutdown_connections 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.686+0000 7f707f200700 1 --2- 172.21.15.102:0/739856296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f70781080c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.686+0000 7f707f200700 1 -- 172.21.15.102:0/739856296 >> 172.21.15.102:0/739856296 conn(0x7f7078103240 msgr2=0x7f7078105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.686+0000 7f707f200700 1 -- 172.21.15.102:0/739856296 shutdown_connections 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.686+0000 7f707f200700 1 -- 172.21.15.102:0/739856296 wait complete. 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.688+0000 7f707f200700 1 Processor -- start 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.688+0000 7f707f200700 1 -- start start 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.689+0000 7f707f200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f7078192e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.689+0000 7f707f200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f707810c250 con 0x7f7078107ca0 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.689+0000 7f7077e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f7078192e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.689+0000 7f7077e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f7078192e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45172/0 (socket says 172.21.15.102:45172) 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.689+0000 7f7077e00700 1 -- 172.21.15.102:0/250516800 learned_addr learned my addr 172.21.15.102:0/250516800 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:38.507 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.690+0000 7f7077e00700 1 -- 172.21.15.102:0/250516800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7078193350 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.690+0000 7f7077e00700 1 --2- 172.21.15.102:0/250516800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f7078192e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7068009fb0 tx=0x7f7068011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.690+0000 7f7074c00700 1 -- 172.21.15.102:0/250516800 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70680036a0 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.690+0000 7f7074c00700 1 -- 172.21.15.102:0/250516800 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f706801a430 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.690+0000 7f7074c00700 1 -- 172.21.15.102:0/250516800 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7068019550 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.690+0000 7f707f200700 1 -- 172.21.15.102:0/250516800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7078191540 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.691+0000 7f707f200700 1 -- 172.21.15.102:0/250516800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7078191a80 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.691+0000 7f7074c00700 1 -- 172.21.15.102:0/250516800 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7068019b90 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.691+0000 7f7074c00700 1 -- 172.21.15.102:0/250516800 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7068024730 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.692+0000 7f707f200700 1 -- 172.21.15.102:0/250516800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7078064a80 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.694+0000 7f7074c00700 1 -- 172.21.15.102:0/250516800 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f7068032070 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.773+0000 7f707f200700 1 -- 172.21.15.102:0/250516800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7078196720 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.774+0000 7f7074c00700 1 -- 172.21.15.102:0/250516800 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f706801a5e0 con 0x7f7078107ca0 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.775+0000 7f707f200700 1 -- 172.21.15.102:0/250516800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 msgr2=0x7f7078192e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:38.508 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.775+0000 7f707f200700 1 --2- 172.21.15.102:0/250516800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f7078192e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7068009fb0 tx=0x7f7068011940 comp rx=0 tx=0).stop 2024-04-26T20:23:38.509 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.775+0000 7f707f200700 1 -- 172.21.15.102:0/250516800 shutdown_connections 2024-04-26T20:23:38.509 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.775+0000 7f707f200700 1 --2- 172.21.15.102:0/250516800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7078107ca0 0x7f7078192e10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:38.509 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.775+0000 7f707f200700 1 -- 172.21.15.102:0/250516800 >> 172.21.15.102:0/250516800 conn(0x7f7078103240 msgr2=0x7f7078104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:38.509 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.776+0000 7f707f200700 1 -- 172.21.15.102:0/250516800 shutdown_connections 2024-04-26T20:23:38.509 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:37.776+0000 7f707f200700 1 -- 172.21.15.102:0/250516800 wait complete. 2024-04-26T20:23:38.509 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (3/15)... 2024-04-26T20:23:38.828 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:38.662+0000 7efecd193200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T20:23:39.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: Activating manager daemon x 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: mgrmap e2: x(active, starting, since 0.0119637s) 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: from='mgr.14100 172.21.15.102:0/1322399398' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: from='mgr.14100 172.21.15.102:0/1322399398' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: from='mgr.14100 172.21.15.102:0/1322399398' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: from='mgr.14100 172.21.15.102:0/1322399398' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: from='mgr.14100 172.21.15.102:0/1322399398' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: Manager daemon x is now available 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: from='mgr.14100 172.21.15.102:0/1322399398' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: from='mgr.14100 172.21.15.102:0/1322399398' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: from='mgr.14100 172.21.15.102:0/1322399398' entity='mgr.x' 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: from='mgr.14100 172.21.15.102:0/1322399398' entity='mgr.x' 2024-04-26T20:23:39.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:38 smithi102 ceph-mon[28094]: from='mgr.14100 172.21.15.102:0/1322399398' entity='mgr.x' 2024-04-26T20:23:40.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:40 smithi102 ceph-mon[28094]: mgrmap e3: x(active, since 1.0209s) 2024-04-26T20:23:41.992 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:41 smithi102 ceph-mon[28094]: mgrmap e4: x(active, since 2s) 2024-04-26T20:23:41.992 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:41 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3415942719' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T20:23:42.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:42.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-26T20:23:42.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T20:23:42.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T20:23:42.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T20:23:42.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T20:23:20.101995+0000", 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-26T20:23:42.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.103+0000 7faea0800700 1 Processor -- start 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.103+0000 7faea0800700 1 -- start start 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.104+0000 7faea0800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9807d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.104+0000 7faea0800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fae9807d880 con 0x7fae9807ee50 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.104+0000 7fae9de00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9807d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.104+0000 7fae9de00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9807d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45252/0 (socket says 172.21.15.102:45252) 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.104+0000 7fae9de00700 1 -- 172.21.15.102:0/3239363943 learned_addr learned my addr 172.21.15.102:0/3239363943 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.105+0000 7fae9de00700 1 -- 172.21.15.102:0/3239363943 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9807da00 con 0x7fae9807ee50 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.105+0000 7fae9de00700 1 --2- 172.21.15.102:0/3239363943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9807d2b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0098a0 tx=0x7fae8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8e30cb6414990b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.105+0000 7fae9ca00700 1 -- 172.21.15.102:0/3239363943 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae8c004160 con 0x7fae9807ee50 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.105+0000 7fae9ca00700 1 -- 172.21.15.102:0/3239363943 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fae8c004300 con 0x7fae9807ee50 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.105+0000 7fae9ca00700 1 -- 172.21.15.102:0/3239363943 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae8c011af0 con 0x7fae9807ee50 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.106+0000 7faea0800700 1 -- 172.21.15.102:0/3239363943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 msgr2=0x7fae9807d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.106+0000 7faea0800700 1 --2- 172.21.15.102:0/3239363943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9807d2b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0098a0 tx=0x7fae8c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.106+0000 7faea0800700 1 -- 172.21.15.102:0/3239363943 shutdown_connections 2024-04-26T20:23:42.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.106+0000 7faea0800700 1 --2- 172.21.15.102:0/3239363943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9807d2b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.106+0000 7faea0800700 1 -- 172.21.15.102:0/3239363943 >> 172.21.15.102:0/3239363943 conn(0x7fae98104370 msgr2=0x7fae98106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.106+0000 7faea0800700 1 -- 172.21.15.102:0/3239363943 shutdown_connections 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.106+0000 7faea0800700 1 -- 172.21.15.102:0/3239363943 wait complete. 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.107+0000 7faea0800700 1 Processor -- start 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.107+0000 7faea0800700 1 -- start start 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.108+0000 7faea0800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.108+0000 7faea0800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fae9810f300 con 0x7fae9807ee50 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.108+0000 7fae9de00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.108+0000 7fae9de00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9819eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45258/0 (socket says 172.21.15.102:45258) 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.108+0000 7fae9de00700 1 -- 172.21.15.102:0/3415942719 learned_addr learned my addr 172.21.15.102:0/3415942719 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.108+0000 7fae9de00700 1 -- 172.21.15.102:0/3415942719 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9819f410 con 0x7fae9807ee50 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.109+0000 7fae9de00700 1 --2- 172.21.15.102:0/3415942719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9819eed0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fae8c003f40 tx=0x7fae8c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.109+0000 7fae92a00700 1 -- 172.21.15.102:0/3415942719 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae8c0038b0 con 0x7fae9807ee50 2024-04-26T20:23:42.304 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.109+0000 7fae92a00700 1 -- 172.21.15.102:0/3415942719 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fae8c003a50 con 0x7fae9807ee50 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.109+0000 7faea0800700 1 -- 172.21.15.102:0/3415942719 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9819f6a0 con 0x7fae9807ee50 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.109+0000 7fae92a00700 1 -- 172.21.15.102:0/3415942719 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae8c01a5c0 con 0x7fae9807ee50 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.109+0000 7faea0800700 1 -- 172.21.15.102:0/3415942719 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae981a2390 con 0x7fae9807ee50 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.110+0000 7faea0800700 1 -- 172.21.15.102:0/3415942719 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae98064a80 con 0x7fae9807ee50 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.111+0000 7fae92a00700 1 -- 172.21.15.102:0/3415942719 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fae8c024430 con 0x7fae9807ee50 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.111+0000 7fae92a00700 1 --2- 172.21.15.102:0/3415942719 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7fae80039560 0x7fae8003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.111+0000 7fae92a00700 1 -- 172.21.15.102:0/3415942719 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fae8c04da20 con 0x7fae9807ee50 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.111+0000 7fae9d400700 1 --2- 172.21.15.102:0/3415942719 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7fae80039560 0x7fae8003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.112+0000 7fae9d400700 1 --2- 172.21.15.102:0/3415942719 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7fae80039560 0x7fae8003ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fae88006fd0 tx=0x7fae88006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.117+0000 7fae92a00700 1 -- 172.21.15.102:0/3415942719 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fae8c04d440 con 0x7fae9807ee50 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.428+0000 7faea0800700 1 -- 172.21.15.102:0/3415942719 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fae981a2560 con 0x7fae9807ee50 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.429+0000 7fae92a00700 1 -- 172.21.15.102:0/3415942719 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fae981a2560 con 0x7fae9807ee50 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.432+0000 7faea0800700 1 -- 172.21.15.102:0/3415942719 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7fae80039560 msgr2=0x7fae8003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.432+0000 7faea0800700 1 --2- 172.21.15.102:0/3415942719 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7fae80039560 0x7fae8003ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fae88006fd0 tx=0x7fae88006e40 comp rx=0 tx=0).stop 2024-04-26T20:23:42.305 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.432+0000 7faea0800700 1 -- 172.21.15.102:0/3415942719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 msgr2=0x7fae9819eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:42.306 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.432+0000 7faea0800700 1 --2- 172.21.15.102:0/3415942719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9819eed0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fae8c003f40 tx=0x7fae8c011890 comp rx=0 tx=0).stop 2024-04-26T20:23:42.306 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.432+0000 7faea0800700 1 -- 172.21.15.102:0/3415942719 shutdown_connections 2024-04-26T20:23:42.306 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.432+0000 7faea0800700 1 --2- 172.21.15.102:0/3415942719 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7fae80039560 0x7fae8003ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:42.306 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.432+0000 7faea0800700 1 --2- 172.21.15.102:0/3415942719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9807ee50 0x7fae9819eed0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:42.306 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.432+0000 7faea0800700 1 -- 172.21.15.102:0/3415942719 >> 172.21.15.102:0/3415942719 conn(0x7fae98104370 msgr2=0x7fae98104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:42.306 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.432+0000 7faea0800700 1 -- 172.21.15.102:0/3415942719 shutdown_connections 2024-04-26T20:23:42.306 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:41.432+0000 7faea0800700 1 -- 172.21.15.102:0/3415942719 wait complete. 2024-04-26T20:23:42.306 INFO:teuthology.orchestra.run.smithi102.stdout:mgr is available 2024-04-26T20:23:43.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:43 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2916300491' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T20:23:43.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:43.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T20:23:43.872 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.876+0000 7feba4000700 1 Processor -- start 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.876+0000 7feba4000700 1 -- start start 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.876+0000 7feba4000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1120b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.876+0000 7feba4000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feb9c1125f0 con 0x7feb9c10fcc0 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.877+0000 7feba1600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1120b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.877+0000 7feba1600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1120b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45272/0 (socket says 172.21.15.102:45272) 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.877+0000 7feba1600700 1 -- 172.21.15.102:0/1392772416 learned_addr learned my addr 172.21.15.102:0/1392772416 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.877+0000 7feba1600700 1 -- 172.21.15.102:0/1392772416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9c112770 con 0x7feb9c10fcc0 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.877+0000 7feba1600700 1 --2- 172.21.15.102:0/1392772416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1120b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb900098a0 tx=0x7feb90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd534057540135c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.878+0000 7feb9be00700 1 -- 172.21.15.102:0/1392772416 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb90004160 con 0x7feb9c10fcc0 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.878+0000 7feb9be00700 1 -- 172.21.15.102:0/1392772416 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7feb90004300 con 0x7feb9c10fcc0 2024-04-26T20:23:43.873 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.878+0000 7feb9be00700 1 -- 172.21.15.102:0/1392772416 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb90011af0 con 0x7feb9c10fcc0 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.878+0000 7feba4000700 1 -- 172.21.15.102:0/1392772416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 msgr2=0x7feb9c1120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.878+0000 7feba4000700 1 --2- 172.21.15.102:0/1392772416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1120b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb900098a0 tx=0x7feb90009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.878+0000 7feba4000700 1 -- 172.21.15.102:0/1392772416 shutdown_connections 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.878+0000 7feba4000700 1 --2- 172.21.15.102:0/1392772416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1120b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.878+0000 7feba4000700 1 -- 172.21.15.102:0/1392772416 >> 172.21.15.102:0/1392772416 conn(0x7feb9c07e370 msgr2=0x7feb9c080790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.879+0000 7feba4000700 1 -- 172.21.15.102:0/1392772416 shutdown_connections 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.879+0000 7feba4000700 1 -- 172.21.15.102:0/1392772416 wait complete. 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.880+0000 7feba4000700 1 Processor -- start 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.880+0000 7feba4000700 1 -- start start 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.880+0000 7feba4000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1a3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.880+0000 7feba4000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feb9c1136d0 con 0x7feb9c10fcc0 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.880+0000 7feba1600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1a3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.880+0000 7feba1600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1a3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45288/0 (socket says 172.21.15.102:45288) 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.880+0000 7feba1600700 1 -- 172.21.15.102:0/2916300491 learned_addr learned my addr 172.21.15.102:0/2916300491 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:43.874 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.881+0000 7feba1600700 1 -- 172.21.15.102:0/2916300491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9c1a3840 con 0x7feb9c10fcc0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.881+0000 7feba1600700 1 --2- 172.21.15.102:0/2916300491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1a3300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feb90011ac0 tx=0x7feb90011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.881+0000 7feb9a000700 1 -- 172.21.15.102:0/2916300491 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb90003970 con 0x7feb9c10fcc0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.881+0000 7feb9a000700 1 -- 172.21.15.102:0/2916300491 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7feb90003b10 con 0x7feb9c10fcc0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.881+0000 7feb9a000700 1 -- 172.21.15.102:0/2916300491 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb90024610 con 0x7feb9c10fcc0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.881+0000 7feba4000700 1 -- 172.21.15.102:0/2916300491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb9c1a3ad0 con 0x7feb9c10fcc0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.881+0000 7feba4000700 1 -- 172.21.15.102:0/2916300491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb9c1a67c0 con 0x7feb9c10fcc0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.882+0000 7feb9a000700 1 -- 172.21.15.102:0/2916300491 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7feb9001a9e0 con 0x7feb9c10fcc0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.882+0000 7feba4000700 1 -- 172.21.15.102:0/2916300491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb9c064a80 con 0x7feb9c10fcc0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.883+0000 7feb9a000700 1 --2- 172.21.15.102:0/2916300491 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7feb840395b0 0x7feb8403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.883+0000 7feb9a000700 1 -- 172.21.15.102:0/2916300491 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7feb9004d860 con 0x7feb9c10fcc0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.883+0000 7feba0c00700 1 --2- 172.21.15.102:0/2916300491 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7feb840395b0 0x7feb8403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.884+0000 7feba0c00700 1 --2- 172.21.15.102:0/2916300491 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7feb840395b0 0x7feb8403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb88006fd0 tx=0x7feb88006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:42.888+0000 7feb9a000700 1 -- 172.21.15.102:0/2916300491 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7feb9000b3e0 con 0x7feb9c10fcc0 2024-04-26T20:23:43.875 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.093+0000 7feba4000700 1 -- 172.21.15.102:0/2916300491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7feb9c1a6990 con 0x7feb9c10fcc0 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.094+0000 7feb9a000700 1 -- 172.21.15.102:0/2916300491 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7feb9c1a6990 con 0x7feb9c10fcc0 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.096+0000 7feba4000700 1 -- 172.21.15.102:0/2916300491 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7feb840395b0 msgr2=0x7feb8403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.096+0000 7feba4000700 1 --2- 172.21.15.102:0/2916300491 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7feb840395b0 0x7feb8403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb88006fd0 tx=0x7feb88006e40 comp rx=0 tx=0).stop 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.096+0000 7feba4000700 1 -- 172.21.15.102:0/2916300491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 msgr2=0x7feb9c1a3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.096+0000 7feba4000700 1 --2- 172.21.15.102:0/2916300491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1a3300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feb90011ac0 tx=0x7feb90011ba0 comp rx=0 tx=0).stop 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.096+0000 7feba4000700 1 -- 172.21.15.102:0/2916300491 shutdown_connections 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.096+0000 7feba4000700 1 --2- 172.21.15.102:0/2916300491 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7feb840395b0 0x7feb8403ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.096+0000 7feba4000700 1 --2- 172.21.15.102:0/2916300491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb9c10fcc0 0x7feb9c1a3300 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.096+0000 7feba4000700 1 -- 172.21.15.102:0/2916300491 >> 172.21.15.102:0/2916300491 conn(0x7feb9c07e370 msgr2=0x7feb9c07ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.097+0000 7feba4000700 1 -- 172.21.15.102:0/2916300491 shutdown_connections 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:43.097+0000 7feba4000700 1 -- 172.21.15.102:0/2916300491 wait complete. 2024-04-26T20:23:43.876 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling cephadm module... 2024-04-26T20:23:45.111 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:44 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2355967432' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T20:23:45.111 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:44 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: ignoring --setuser ceph since I am not root 2024-04-26T20:23:45.111 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:44 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: ignoring --setgroup ceph since I am not root 2024-04-26T20:23:45.112 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:45 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:45.110+0000 7fe0411e5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T20:23:45.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.472+0000 7f3784e00700 1 Processor -- start 2024-04-26T20:23:45.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.472+0000 7f3784e00700 1 -- start start 2024-04-26T20:23:45.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.473+0000 7f3784e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f3780109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:45.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.473+0000 7f3784e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3780109760 con 0x7f3780108d70 2024-04-26T20:23:45.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.473+0000 7f377de00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f3780109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:45.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.473+0000 7f377de00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f3780109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45290/0 (socket says 172.21.15.102:45290) 2024-04-26T20:23:45.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.473+0000 7f377de00700 1 -- 172.21.15.102:0/3411073708 learned_addr learned my addr 172.21.15.102:0/3411073708 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:45.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.474+0000 7f377de00700 1 -- 172.21.15.102:0/3411073708 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3780109f70 con 0x7f3780108d70 2024-04-26T20:23:45.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.474+0000 7f377de00700 1 --2- 172.21.15.102:0/3411073708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f3780109190 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3770009e80 tx=0x7f377000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24be5f3409bfca05 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:45.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.474+0000 7f377d400700 1 -- 172.21.15.102:0/3411073708 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3770017070 con 0x7f3780108d70 2024-04-26T20:23:45.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.474+0000 7f377d400700 1 -- 172.21.15.102:0/3411073708 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f377000b8c0 con 0x7f3780108d70 2024-04-26T20:23:45.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.474+0000 7f377d400700 1 -- 172.21.15.102:0/3411073708 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3770004490 con 0x7f3780108d70 2024-04-26T20:23:45.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.475+0000 7f3784e00700 1 -- 172.21.15.102:0/3411073708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 msgr2=0x7f3780109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:45.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.475+0000 7f3784e00700 1 --2- 172.21.15.102:0/3411073708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f3780109190 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3770009e80 tx=0x7f377000b330 comp rx=0 tx=0).stop 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.475+0000 7f3784e00700 1 -- 172.21.15.102:0/3411073708 shutdown_connections 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.475+0000 7f3784e00700 1 --2- 172.21.15.102:0/3411073708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f3780109190 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.475+0000 7f3784e00700 1 -- 172.21.15.102:0/3411073708 >> 172.21.15.102:0/3411073708 conn(0x7f3780104370 msgr2=0x7f3780106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.475+0000 7f3784e00700 1 -- 172.21.15.102:0/3411073708 shutdown_connections 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.475+0000 7f3784e00700 1 -- 172.21.15.102:0/3411073708 wait complete. 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.477+0000 7f3784e00700 1 Processor -- start 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.477+0000 7f3784e00700 1 -- start start 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.477+0000 7f3784e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f378019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.477+0000 7f3784e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f378010d320 con 0x7f3780108d70 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.477+0000 7f377de00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f378019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.477+0000 7f377de00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f378019ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45300/0 (socket says 172.21.15.102:45300) 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.477+0000 7f377de00700 1 -- 172.21.15.102:0/2355967432 learned_addr learned my addr 172.21.15.102:0/2355967432 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.478+0000 7f377de00700 1 -- 172.21.15.102:0/2355967432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378019d200 con 0x7f3780108d70 2024-04-26T20:23:45.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.478+0000 7f377de00700 1 --2- 172.21.15.102:0/2355967432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f378019ccc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3770004a60 tx=0x7f377000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.478+0000 7f3777400700 1 -- 172.21.15.102:0/2355967432 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3770020020 con 0x7f3780108d70 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.478+0000 7f3777400700 1 -- 172.21.15.102:0/2355967432 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3770003c60 con 0x7f3780108d70 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.478+0000 7f3784e00700 1 -- 172.21.15.102:0/2355967432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378019d490 con 0x7f3780108d70 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.478+0000 7f3777400700 1 -- 172.21.15.102:0/2355967432 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3770017040 con 0x7f3780108d70 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.479+0000 7f3784e00700 1 -- 172.21.15.102:0/2355967432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37801a0180 con 0x7f3780108d70 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.479+0000 7f3777400700 1 -- 172.21.15.102:0/2355967432 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f3770023020 con 0x7f3780108d70 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.480+0000 7f3784e00700 1 -- 172.21.15.102:0/2355967432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3780064a80 con 0x7f3780108d70 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.480+0000 7f3777400700 1 --2- 172.21.15.102:0/2355967432 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f3764039560 0x7f376403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.480+0000 7f3777400700 1 -- 172.21.15.102:0/2355967432 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f377000f050 con 0x7f3780108d70 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.480+0000 7f3775400700 1 --2- 172.21.15.102:0/2355967432 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f3764039560 0x7f376403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:45.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.481+0000 7f3775400700 1 --2- 172.21.15.102:0/2355967432 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f3764039560 0x7f376403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3768006fd0 tx=0x7f3768006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:45.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.486+0000 7f3777400700 1 -- 172.21.15.102:0/2355967432 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f377002a020 con 0x7f3780108d70 2024-04-26T20:23:45.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.768+0000 7f3784e00700 1 -- 172.21.15.102:0/2355967432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f37801a0870 con 0x7f3780108d70 2024-04-26T20:23:45.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.821+0000 7f3777400700 1 -- 172.21.15.102:0/2355967432 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f377002c300 con 0x7f3780108d70 2024-04-26T20:23:45.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.825+0000 7f3784e00700 1 -- 172.21.15.102:0/2355967432 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f3764039560 msgr2=0x7f376403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:45.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.825+0000 7f3784e00700 1 --2- 172.21.15.102:0/2355967432 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f3764039560 0x7f376403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3768006fd0 tx=0x7f3768006e40 comp rx=0 tx=0).stop 2024-04-26T20:23:45.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.825+0000 7f3784e00700 1 -- 172.21.15.102:0/2355967432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 msgr2=0x7f378019ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:45.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.825+0000 7f3777400700 1 -- 172.21.15.102:0/2355967432 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f3770012950 con 0x7f3780108d70 2024-04-26T20:23:45.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.825+0000 7f3784e00700 1 --2- 172.21.15.102:0/2355967432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f378019ccc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3770004a60 tx=0x7f377000bd50 comp rx=0 tx=0).stop 2024-04-26T20:23:45.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.826+0000 7f3784e00700 1 -- 172.21.15.102:0/2355967432 shutdown_connections 2024-04-26T20:23:45.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.826+0000 7f3784e00700 1 --2- 172.21.15.102:0/2355967432 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f3764039560 0x7f376403ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:45.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.826+0000 7f3784e00700 1 --2- 172.21.15.102:0/2355967432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3780108d70 0x7f378019ccc0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:45.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.826+0000 7f3784e00700 1 -- 172.21.15.102:0/2355967432 >> 172.21.15.102:0/2355967432 conn(0x7f3780104370 msgr2=0x7f3780104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:45.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.827+0000 7f3784e00700 1 -- 172.21.15.102:0/2355967432 shutdown_connections 2024-04-26T20:23:45.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:44.827+0000 7f3784e00700 1 -- 172.21.15.102:0/2355967432 wait complete. 2024-04-26T20:23:45.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:45 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:45.361+0000 7fe0411e5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:23:45.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:45 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:45.492+0000 7fe0411e5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T20:23:45.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:45 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2355967432' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T20:23:45.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:45 smithi102 ceph-mon[28094]: mgrmap e5: x(active, since 6s) 2024-04-26T20:23:45.997 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:45 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:45.898+0000 7fe0411e5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T20:23:46.360 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:46 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:46.011+0000 7fe0411e5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T20:23:47.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:46 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1593317464' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T20:23:47.065 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-26T20:23:47.065 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T20:23:47.065 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T20:23:47.065 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T20:23:47.065 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T20:23:47.065 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-26T20:23:47.065 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.150+0000 7f1781600700 1 Processor -- start 2024-04-26T20:23:47.071 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.150+0000 7f1781600700 1 -- start start 2024-04-26T20:23:47.071 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.150+0000 7f1781600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c07d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:47.071 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.150+0000 7f1781600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f177c07d900 con 0x7f177c07eed0 2024-04-26T20:23:47.071 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.151+0000 7f177aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c07d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:47.071 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.151+0000 7f177aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c07d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33952/0 (socket says 172.21.15.102:33952) 2024-04-26T20:23:47.071 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.151+0000 7f177aa00700 1 -- 172.21.15.102:0/3713130140 learned_addr learned my addr 172.21.15.102:0/3713130140 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:47.071 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.151+0000 7f177aa00700 1 -- 172.21.15.102:0/3713130140 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c07da80 con 0x7f177c07eed0 2024-04-26T20:23:47.071 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.152+0000 7f177aa00700 1 --2- 172.21.15.102:0/3713130140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c07d330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f176c0098a0 tx=0x7f176c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20ec374064b9f5d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:47.071 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.152+0000 7f1779600700 1 -- 172.21.15.102:0/3713130140 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f176c004160 con 0x7f177c07eed0 2024-04-26T20:23:47.071 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.152+0000 7f1779600700 1 -- 172.21.15.102:0/3713130140 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f176c004300 con 0x7f177c07eed0 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.152+0000 7f1779600700 1 -- 172.21.15.102:0/3713130140 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f176c011af0 con 0x7f177c07eed0 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.152+0000 7f1781600700 1 -- 172.21.15.102:0/3713130140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 msgr2=0x7f177c07d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.152+0000 7f1781600700 1 --2- 172.21.15.102:0/3713130140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c07d330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f176c0098a0 tx=0x7f176c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.153+0000 7f1781600700 1 -- 172.21.15.102:0/3713130140 shutdown_connections 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.153+0000 7f1781600700 1 --2- 172.21.15.102:0/3713130140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c07d330 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.153+0000 7f1781600700 1 -- 172.21.15.102:0/3713130140 >> 172.21.15.102:0/3713130140 conn(0x7f177c104370 msgr2=0x7f177c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.153+0000 7f1781600700 1 -- 172.21.15.102:0/3713130140 shutdown_connections 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.153+0000 7f1781600700 1 -- 172.21.15.102:0/3713130140 wait complete. 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.154+0000 7f1781600700 1 Processor -- start 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.154+0000 7f1781600700 1 -- start start 2024-04-26T20:23:47.072 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.154+0000 7f1781600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c1a32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.154+0000 7f1781600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f177c10af20 con 0x7f177c07eed0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.155+0000 7f177aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c1a32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.155+0000 7f177aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c1a32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33964/0 (socket says 172.21.15.102:33964) 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.155+0000 7f177aa00700 1 -- 172.21.15.102:0/1593317464 learned_addr learned my addr 172.21.15.102:0/1593317464 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.155+0000 7f177aa00700 1 -- 172.21.15.102:0/1593317464 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c1a3810 con 0x7f177c07eed0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.155+0000 7f177aa00700 1 --2- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c1a32d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f176c003f40 tx=0x7f176c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.156+0000 7f1773400700 1 -- 172.21.15.102:0/1593317464 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f176c0038b0 con 0x7f177c07eed0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.156+0000 7f1773400700 1 -- 172.21.15.102:0/1593317464 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f176c003a50 con 0x7f177c07eed0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.156+0000 7f1773400700 1 -- 172.21.15.102:0/1593317464 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f176c01a5c0 con 0x7f177c07eed0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.156+0000 7f1781600700 1 -- 172.21.15.102:0/1593317464 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f177c1a3aa0 con 0x7f177c07eed0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.156+0000 7f1781600700 1 -- 172.21.15.102:0/1593317464 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f177c1a6790 con 0x7f177c07eed0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.157+0000 7f1773400700 1 -- 172.21.15.102:0/1593317464 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f176c024430 con 0x7f177c07eed0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.157+0000 7f1781600700 1 -- 172.21.15.102:0/1593317464 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f177c064a80 con 0x7f177c07eed0 2024-04-26T20:23:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.158+0000 7f1773400700 1 --2- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f17600395b0 0x7f176003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.158+0000 7f1773400700 1 -- 172.21.15.102:0/1593317464 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f176c04daf0 con 0x7f177c07eed0 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.158+0000 7f177a000700 1 -- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f17600395b0 msgr2=0x7f176003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3976810658 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.158+0000 7f177a000700 1 --2- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f17600395b0 0x7f176003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.163+0000 7f1773400700 1 -- 172.21.15.102:0/1593317464 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f176c04d510 con 0x7f177c07eed0 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.359+0000 7f177a000700 1 -- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f17600395b0 msgr2=0x7f176003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3976810658 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.359+0000 7f177a000700 1 --2- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f17600395b0 0x7f176003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.448+0000 7f1781600700 1 -- 172.21.15.102:0/1593317464 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f177c1a6960 con 0x7f177c07eed0 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.449+0000 7f1773400700 1 -- 172.21.15.102:0/1593317464 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f177c1a6960 con 0x7f177c07eed0 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.451+0000 7f1781600700 1 -- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f17600395b0 msgr2=0x7f176003ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.451+0000 7f1781600700 1 --2- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f17600395b0 0x7f176003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:47.074 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.451+0000 7f1781600700 1 -- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 msgr2=0x7f177c1a32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:47.075 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.451+0000 7f1781600700 1 --2- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c1a32d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f176c003f40 tx=0x7f176c011890 comp rx=0 tx=0).stop 2024-04-26T20:23:47.075 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.452+0000 7f1781600700 1 -- 172.21.15.102:0/1593317464 shutdown_connections 2024-04-26T20:23:47.075 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.452+0000 7f1781600700 1 --2- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f17600395b0 0x7f176003ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:47.075 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.452+0000 7f1781600700 1 --2- 172.21.15.102:0/1593317464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f177c07eed0 0x7f177c1a32d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:47.075 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.452+0000 7f1781600700 1 -- 172.21.15.102:0/1593317464 >> 172.21.15.102:0/1593317464 conn(0x7f177c104370 msgr2=0x7f177c106760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:47.075 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.452+0000 7f1781600700 1 -- 172.21.15.102:0/1593317464 shutdown_connections 2024-04-26T20:23:47.075 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:46.452+0000 7f1781600700 1 -- 172.21.15.102:0/1593317464 wait complete. 2024-04-26T20:23:47.075 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-04-26T20:23:47.075 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 5... 2024-04-26T20:23:47.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:47 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:47.571+0000 7fe0411e5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T20:23:48.018 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:47 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:47.797+0000 7fe0411e5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T20:23:48.394 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:48 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:48.016+0000 7fe0411e5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T20:23:48.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:48 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:48.393+0000 7fe0411e5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T20:23:48.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:48 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:48.514+0000 7fe0411e5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T20:23:49.022 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:48 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:48.767+0000 7fe0411e5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T20:23:49.398 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:49 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:49.020+0000 7fe0411e5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:23:50.148 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:49 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:49.802+0000 7fe0411e5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T20:23:50.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:50 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:50.345+0000 7fe0411e5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T20:23:50.649 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:50 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:50.474+0000 7fe0411e5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T20:23:51.148 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:50 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:50.741+0000 7fe0411e5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T20:23:51.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:51.380+0000 7fe0411e5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T20:23:51.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:51.503+0000 7fe0411e5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T20:23:52.125 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:51.822+0000 7fe0411e5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T20:23:52.126 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:51.997+0000 7fe0411e5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T20:23:52.126 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:52 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:52.124+0000 7fe0411e5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T20:23:52.398 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:52 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:52.273+0000 7fe0411e5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T20:23:52.898 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:52 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:52.638+0000 7fe0411e5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T20:23:53.490 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:53.395+0000 7fe0411e5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T20:23:53.747 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:23:53.691+0000 7fe0411e5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T20:23:54.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-mon[28094]: Active manager daemon x restarted 2024-04-26T20:23:54.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-mon[28094]: Activating manager daemon x 2024-04-26T20:23:54.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-mon[28094]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T20:23:54.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-mon[28094]: mgrmap e6: x(active, starting, since 0.00908289s) 2024-04-26T20:23:54.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:23:54.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T20:23:54.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T20:23:54.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T20:23:54.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T20:23:54.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:53 smithi102 ceph-mon[28094]: Manager daemon x is now available 2024-04-26T20:23:55.095 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:54 smithi102 ceph-mon[28094]: Found migration_current of "None". Setting to last migration. 2024-04-26T20:23:55.096 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:54 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:23:55.096 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:54 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:23:55.096 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:54 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:23:55.096 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:54 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:23:55.096 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:54 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T20:23:55.096 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:54 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T20:23:55.096 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:54 smithi102 ceph-mon[28094]: mgrmap e7: x(active, since 1.01475s) 2024-04-26T20:23:55.405 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.651+0000 7f1751000700 1 Processor -- start 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.651+0000 7f1751000700 1 -- start start 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.652+0000 7f1751000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c10bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.652+0000 7f1751000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f174c07ce40 con 0x7f174c109870 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.652+0000 7f1749e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c10bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.652+0000 7f1749e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c10bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33974/0 (socket says 172.21.15.102:33974) 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.652+0000 7f1749e00700 1 -- 172.21.15.102:0/2777340085 learned_addr learned my addr 172.21.15.102:0/2777340085 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:55.406 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.652+0000 7f1749e00700 1 -- 172.21.15.102:0/2777340085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c10c1a0 con 0x7f174c109870 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.653+0000 7f1749e00700 1 --2- 172.21.15.102:0/2777340085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c10bc60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f173c0098a0 tx=0x7f173c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134a3cf890a91ec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.653+0000 7f1748a00700 1 -- 172.21.15.102:0/2777340085 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f173c004160 con 0x7f174c109870 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.653+0000 7f1748a00700 1 -- 172.21.15.102:0/2777340085 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f173c004300 con 0x7f174c109870 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.653+0000 7f1748a00700 1 -- 172.21.15.102:0/2777340085 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f173c011af0 con 0x7f174c109870 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.654+0000 7f1751000700 1 -- 172.21.15.102:0/2777340085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 msgr2=0x7f174c10bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.654+0000 7f1751000700 1 --2- 172.21.15.102:0/2777340085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c10bc60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f173c0098a0 tx=0x7f173c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.654+0000 7f1751000700 1 -- 172.21.15.102:0/2777340085 shutdown_connections 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.654+0000 7f1751000700 1 --2- 172.21.15.102:0/2777340085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c10bc60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.654+0000 7f1751000700 1 -- 172.21.15.102:0/2777340085 >> 172.21.15.102:0/2777340085 conn(0x7f174c103240 msgr2=0x7f174c105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.654+0000 7f1751000700 1 -- 172.21.15.102:0/2777340085 shutdown_connections 2024-04-26T20:23:55.407 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.654+0000 7f1751000700 1 -- 172.21.15.102:0/2777340085 wait complete. 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.655+0000 7f1751000700 1 Processor -- start 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.655+0000 7f1751000700 1 -- start start 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.655+0000 7f1751000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c1090f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.655+0000 7f1751000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f174c10d110 con 0x7f174c109870 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.656+0000 7f1749e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c1090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.656+0000 7f1749e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c1090f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33984/0 (socket says 172.21.15.102:33984) 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.656+0000 7f1749e00700 1 -- 172.21.15.102:0/414746655 learned_addr learned my addr 172.21.15.102:0/414746655 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.656+0000 7f1749e00700 1 -- 172.21.15.102:0/414746655 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c109630 con 0x7f174c109870 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.656+0000 7f1749e00700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c1090f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f173c011ac0 tx=0x7f173c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.657+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f173c003970 con 0x7f174c109870 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.657+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f173c003b10 con 0x7f174c109870 2024-04-26T20:23:55.408 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.657+0000 7f1751000700 1 -- 172.21.15.102:0/414746655 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174c107820 con 0x7f174c109870 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.657+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f173c024610 con 0x7f174c109870 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.657+0000 7f1751000700 1 -- 172.21.15.102:0/414746655 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f174c107d60 con 0x7f174c109870 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.658+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f173c024840 con 0x7f174c109870 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.659+0000 7f1742a00700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 0x7f173003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.659+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 --> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f173003c230 con 0x7f1730039660 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.659+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f173c04da20 con 0x7f174c109870 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.659+0000 7f1749400700 1 -- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 msgr2=0x7f173003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3976810658 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.659+0000 7f1749400700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 0x7f173003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.860+0000 7f1749400700 1 -- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 msgr2=0x7f173003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3976810658 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:47.860+0000 7f1749400700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 0x7f173003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:48.261+0000 7f1749400700 1 -- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 msgr2=0x7f173003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3976810658 2024-04-26T20:23:55.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:48.261+0000 7f1749400700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 0x7f173003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:49.062+0000 7f1749400700 1 -- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 msgr2=0x7f173003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3976810658 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:49.062+0000 7f1749400700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 0x7f173003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:50.664+0000 7f1749400700 1 -- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 msgr2=0x7f173003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/3976810658 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:50.664+0000 7f1749400700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 0x7f173003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:53.699+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f173c024b30 con 0x7f174c109870 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:53.699+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 msgr2=0x7f173003bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:53.699+0000 7f1742a00700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 0x7f173003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.703+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f173c04faf0 con 0x7f174c109870 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.704+0000 7f1742a00700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f173003cf50 0x7f173003f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.704+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 --> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f173c020e50 con 0x7f173003cf50 2024-04-26T20:23:55.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.707+0000 7f1749400700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f173003cf50 0x7f173003f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.709+0000 7f1749400700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f173003cf50 0x7f173003f340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1738003a10 tx=0x7f17380092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.710+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 <== mgr.14120 v2:172.21.15.102:6800/1441707916 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f173c020e50 con 0x7f173003cf50 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.720+0000 7f1751000700 1 -- 172.21.15.102:0/414746655 --> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f174c06a540 con 0x7f173003cf50 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.721+0000 7f1742a00700 1 -- 172.21.15.102:0/414746655 <== mgr.14120 v2:172.21.15.102:6800/1441707916 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f174c06a540 con 0x7f173003cf50 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.721+0000 7f1751000700 1 -- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f173003cf50 msgr2=0x7f173003f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.721+0000 7f1751000700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f173003cf50 0x7f173003f340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1738003a10 tx=0x7f17380092b0 comp rx=0 tx=0).stop 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.721+0000 7f1751000700 1 -- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 msgr2=0x7f174c1090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.721+0000 7f1751000700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c1090f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f173c011ac0 tx=0x7f173c011ba0 comp rx=0 tx=0).stop 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.722+0000 7f1751000700 1 -- 172.21.15.102:0/414746655 shutdown_connections 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.722+0000 7f1751000700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f173003cf50 0x7f173003f340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.722+0000 7f1751000700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f174c109870 0x7f174c1090f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.722+0000 7f1751000700 1 --2- 172.21.15.102:0/414746655 >> [v2:172.21.15.102:6800/3976810658,v1:172.21.15.102:6801/3976810658] conn(0x7f1730039660 0x7f173003bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:55.411 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.722+0000 7f1751000700 1 -- 172.21.15.102:0/414746655 >> 172.21.15.102:0/414746655 conn(0x7f174c103240 msgr2=0x7f174c103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:55.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.722+0000 7f1751000700 1 -- 172.21.15.102:0/414746655 shutdown_connections 2024-04-26T20:23:55.412 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:54.722+0000 7f1751000700 1 -- 172.21.15.102:0/414746655 wait complete. 2024-04-26T20:23:55.412 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 5 is available 2024-04-26T20:23:55.412 INFO:teuthology.orchestra.run.smithi102.stdout:Setting orchestrator backend to cephadm... 2024-04-26T20:23:56.014 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:55 smithi102 ceph-mon[28094]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T20:23:56.014 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:55 smithi102 ceph-mon[28094]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T20:23:56.014 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:55 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:23:56.014 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:55 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:23:56.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:56 smithi102 ceph-mon[28094]: mgrmap e8: x(active, since 2s) 2024-04-26T20:23:56.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:56 smithi102 ceph-mon[28094]: [26/Apr/2024:20:23:55] ENGINE Bus STARTING 2024-04-26T20:23:56.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:56 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:23:56.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:56 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:23:56.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:56 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:23:56.940 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.012+0000 7f62f4600700 1 Processor -- start 2024-04-26T20:23:56.940 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.012+0000 7f62f4600700 1 -- start start 2024-04-26T20:23:56.940 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.013+0000 7f62f4600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec10bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:56.940 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.013+0000 7f62f4600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec07cea0 con 0x7f62ec109810 2024-04-26T20:23:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.013+0000 7f62f1c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec10bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.013+0000 7f62f1c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec10bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36846/0 (socket says 172.21.15.102:36846) 2024-04-26T20:23:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.013+0000 7f62f1c00700 1 -- 172.21.15.102:0/2049373735 learned_addr learned my addr 172.21.15.102:0/2049373735 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.014+0000 7f62f1c00700 1 -- 172.21.15.102:0/2049373735 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec10c140 con 0x7f62ec109810 2024-04-26T20:23:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.014+0000 7f62f1c00700 1 --2- 172.21.15.102:0/2049373735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec10bc00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62e00098a0 tx=0x7f62e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d927ea3e344549 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.014+0000 7f62ebe00700 1 -- 172.21.15.102:0/2049373735 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62e0004160 con 0x7f62ec109810 2024-04-26T20:23:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.014+0000 7f62ebe00700 1 -- 172.21.15.102:0/2049373735 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f62e0004300 con 0x7f62ec109810 2024-04-26T20:23:56.943 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.014+0000 7f62ebe00700 1 -- 172.21.15.102:0/2049373735 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62e0011af0 con 0x7f62ec109810 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.015+0000 7f62f4600700 1 -- 172.21.15.102:0/2049373735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 msgr2=0x7f62ec10bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.015+0000 7f62f4600700 1 --2- 172.21.15.102:0/2049373735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec10bc00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62e00098a0 tx=0x7f62e0009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.015+0000 7f62f4600700 1 -- 172.21.15.102:0/2049373735 shutdown_connections 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.015+0000 7f62f4600700 1 --2- 172.21.15.102:0/2049373735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec10bc00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.015+0000 7f62f4600700 1 -- 172.21.15.102:0/2049373735 >> 172.21.15.102:0/2049373735 conn(0x7f62ec103240 msgr2=0x7f62ec105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.015+0000 7f62f4600700 1 -- 172.21.15.102:0/2049373735 shutdown_connections 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.015+0000 7f62f4600700 1 -- 172.21.15.102:0/2049373735 wait complete. 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.016+0000 7f62f4600700 1 Processor -- start 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.016+0000 7f62f4600700 1 -- start start 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.017+0000 7f62f4600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec109090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.017+0000 7f62f4600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec10d110 con 0x7f62ec109810 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.017+0000 7f62f1c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec109090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:56.944 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.017+0000 7f62f1c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec109090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36854/0 (socket says 172.21.15.102:36854) 2024-04-26T20:23:56.945 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.017+0000 7f62f1c00700 1 -- 172.21.15.102:0/3863498559 learned_addr learned my addr 172.21.15.102:0/3863498559 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:56.945 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.017+0000 7f62f1c00700 1 -- 172.21.15.102:0/3863498559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec1095d0 con 0x7f62ec109810 2024-04-26T20:23:56.945 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.018+0000 7f62f1c00700 1 --2- 172.21.15.102:0/3863498559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec109090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f62e0011ac0 tx=0x7f62e0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:56.945 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.018+0000 7f62ea000700 1 -- 172.21.15.102:0/3863498559 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62e0003970 con 0x7f62ec109810 2024-04-26T20:23:56.948 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.018+0000 7f62ea000700 1 -- 172.21.15.102:0/3863498559 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f62e0003b10 con 0x7f62ec109810 2024-04-26T20:23:56.948 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.018+0000 7f62ea000700 1 -- 172.21.15.102:0/3863498559 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62e0024610 con 0x7f62ec109810 2024-04-26T20:23:56.948 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.018+0000 7f62f4600700 1 -- 172.21.15.102:0/3863498559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec1077c0 con 0x7f62ec109810 2024-04-26T20:23:56.948 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.018+0000 7f62f4600700 1 -- 172.21.15.102:0/3863498559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec107d00 con 0x7f62ec109810 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.019+0000 7f62ea000700 1 -- 172.21.15.102:0/3863498559 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f62e001a9e0 con 0x7f62ec109810 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.019+0000 7f62f4600700 1 -- 172.21.15.102:0/3863498559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ec064a80 con 0x7f62ec109810 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.020+0000 7f62ea000700 1 --2- 172.21.15.102:0/3863498559 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f62d4039600 0x7f62d403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.020+0000 7f62ea000700 1 -- 172.21.15.102:0/3863498559 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f62e004d940 con 0x7f62ec109810 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.020+0000 7f62f1200700 1 --2- 172.21.15.102:0/3863498559 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f62d4039600 0x7f62d403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.021+0000 7f62f1200700 1 --2- 172.21.15.102:0/3863498559 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f62d4039600 0x7f62d403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f62d8006fd0 tx=0x7f62d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.026+0000 7f62ea000700 1 -- 172.21.15.102:0/3863498559 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f62e000b3e0 con 0x7f62ec109810 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.253+0000 7f62f4600700 1 -- 172.21.15.102:0/3863498559 --> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f62ec104490 con 0x7f62d4039600 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.265+0000 7f62ea000700 1 -- 172.21.15.102:0/3863498559 <== mgr.14120 v2:172.21.15.102:6800/1441707916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f62ec104490 con 0x7f62d4039600 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.268+0000 7f62f4600700 1 -- 172.21.15.102:0/3863498559 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f62d4039600 msgr2=0x7f62d403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.268+0000 7f62f4600700 1 --2- 172.21.15.102:0/3863498559 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f62d4039600 0x7f62d403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f62d8006fd0 tx=0x7f62d8006e40 comp rx=0 tx=0).stop 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.268+0000 7f62f4600700 1 -- 172.21.15.102:0/3863498559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 msgr2=0x7f62ec109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.268+0000 7f62f4600700 1 --2- 172.21.15.102:0/3863498559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec109090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f62e0011ac0 tx=0x7f62e0011ba0 comp rx=0 tx=0).stop 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.268+0000 7f62f4600700 1 -- 172.21.15.102:0/3863498559 shutdown_connections 2024-04-26T20:23:56.949 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.268+0000 7f62f4600700 1 --2- 172.21.15.102:0/3863498559 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f62d4039600 0x7f62d403bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:56.950 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.268+0000 7f62f4600700 1 --2- 172.21.15.102:0/3863498559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62ec109810 0x7f62ec109090 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:56.950 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.268+0000 7f62f4600700 1 -- 172.21.15.102:0/3863498559 >> 172.21.15.102:0/3863498559 conn(0x7f62ec103240 msgr2=0x7f62ec103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:56.950 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.268+0000 7f62f4600700 1 -- 172.21.15.102:0/3863498559 shutdown_connections 2024-04-26T20:23:56.950 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:56.269+0000 7f62f4600700 1 -- 172.21.15.102:0/3863498559 wait complete. 2024-04-26T20:23:58.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:57 smithi102 ceph-mon[28094]: [26/Apr/2024:20:23:55] ENGINE Serving on https://172.21.15.102:7150 2024-04-26T20:23:58.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:57 smithi102 ceph-mon[28094]: [26/Apr/2024:20:23:56] ENGINE Serving on http://172.21.15.102:8765 2024-04-26T20:23:58.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:57 smithi102 ceph-mon[28094]: [26/Apr/2024:20:23:56] ENGINE Bus STARTED 2024-04-26T20:23:58.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:57 smithi102 ceph-mon[28094]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:23:58.463 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T20:23:58.463 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.523+0000 7f7d20200700 1 Processor -- start 2024-04-26T20:23:58.464 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.523+0000 7f7d20200700 1 -- start start 2024-04-26T20:23:58.464 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.523+0000 7f7d20200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1810b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:58.464 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.523+0000 7f7d20200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1810b960 con 0x7f7d1810af70 2024-04-26T20:23:58.464 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.524+0000 7f7d1d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1810b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:58.465 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.524+0000 7f7d1d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1810b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36864/0 (socket says 172.21.15.102:36864) 2024-04-26T20:23:58.465 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.524+0000 7f7d1d800700 1 -- 172.21.15.102:0/3100112836 learned_addr learned my addr 172.21.15.102:0/3100112836 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:58.465 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.524+0000 7f7d1d800700 1 -- 172.21.15.102:0/3100112836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1810c170 con 0x7f7d1810af70 2024-04-26T20:23:58.465 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.524+0000 7f7d1d800700 1 --2- 172.21.15.102:0/3100112836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1810b390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0098a0 tx=0x7f7d0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440c75ba0bd9822a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:58.465 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.525+0000 7f7d17e00700 1 -- 172.21.15.102:0/3100112836 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d0c004160 con 0x7f7d1810af70 2024-04-26T20:23:58.466 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.525+0000 7f7d17e00700 1 -- 172.21.15.102:0/3100112836 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7d0c004300 con 0x7f7d1810af70 2024-04-26T20:23:58.466 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.525+0000 7f7d17e00700 1 -- 172.21.15.102:0/3100112836 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d0c011af0 con 0x7f7d1810af70 2024-04-26T20:23:58.466 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.525+0000 7f7d20200700 1 -- 172.21.15.102:0/3100112836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 msgr2=0x7f7d1810b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:58.466 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.525+0000 7f7d20200700 1 --2- 172.21.15.102:0/3100112836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1810b390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0098a0 tx=0x7f7d0c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:23:58.466 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.526+0000 7f7d20200700 1 -- 172.21.15.102:0/3100112836 shutdown_connections 2024-04-26T20:23:58.467 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.526+0000 7f7d20200700 1 --2- 172.21.15.102:0/3100112836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1810b390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:58.467 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.526+0000 7f7d20200700 1 -- 172.21.15.102:0/3100112836 >> 172.21.15.102:0/3100112836 conn(0x7f7d1807e370 msgr2=0x7f7d1807e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:58.467 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.526+0000 7f7d20200700 1 -- 172.21.15.102:0/3100112836 shutdown_connections 2024-04-26T20:23:58.467 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.526+0000 7f7d20200700 1 -- 172.21.15.102:0/3100112836 wait complete. 2024-04-26T20:23:58.467 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.527+0000 7f7d20200700 1 Processor -- start 2024-04-26T20:23:58.468 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.527+0000 7f7d20200700 1 -- start start 2024-04-26T20:23:58.468 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.527+0000 7f7d20200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:58.468 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.527+0000 7f7d20200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1810f520 con 0x7f7d1810af70 2024-04-26T20:23:58.469 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.527+0000 7f7d1d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:58.469 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.528+0000 7f7d1d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1819ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36874/0 (socket says 172.21.15.102:36874) 2024-04-26T20:23:58.469 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.528+0000 7f7d1d800700 1 -- 172.21.15.102:0/2190266339 learned_addr learned my addr 172.21.15.102:0/2190266339 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:23:58.469 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.528+0000 7f7d1d800700 1 -- 172.21.15.102:0/2190266339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1819f490 con 0x7f7d1810af70 2024-04-26T20:23:58.470 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.528+0000 7f7d1d800700 1 --2- 172.21.15.102:0/2190266339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1819ef50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0117b0 tx=0x7f7d0c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:58.470 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.528+0000 7f7d16000700 1 -- 172.21.15.102:0/2190266339 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d0c003af0 con 0x7f7d1810af70 2024-04-26T20:23:58.470 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.528+0000 7f7d16000700 1 -- 172.21.15.102:0/2190266339 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7d0c020070 con 0x7f7d1810af70 2024-04-26T20:23:58.470 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.528+0000 7f7d16000700 1 -- 172.21.15.102:0/2190266339 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d0c019630 con 0x7f7d1810af70 2024-04-26T20:23:58.470 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.528+0000 7f7d20200700 1 -- 172.21.15.102:0/2190266339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1819f720 con 0x7f7d1810af70 2024-04-26T20:23:58.470 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.529+0000 7f7d20200700 1 -- 172.21.15.102:0/2190266339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d181a2410 con 0x7f7d1810af70 2024-04-26T20:23:58.470 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.530+0000 7f7d20200700 1 -- 172.21.15.102:0/2190266339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d1806a540 con 0x7f7d1810af70 2024-04-26T20:23:58.470 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.530+0000 7f7d16000700 1 -- 172.21.15.102:0/2190266339 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7d0c01a9c0 con 0x7f7d1810af70 2024-04-26T20:23:58.471 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.531+0000 7f7d16000700 1 --2- 172.21.15.102:0/2190266339 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f7d00039600 0x7f7d0003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:23:58.471 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.531+0000 7f7d16000700 1 -- 172.21.15.102:0/2190266339 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7d0c04dbb0 con 0x7f7d1810af70 2024-04-26T20:23:58.472 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.531+0000 7f7d1ce00700 1 --2- 172.21.15.102:0/2190266339 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f7d00039600 0x7f7d0003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:23:58.472 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.531+0000 7f7d1ce00700 1 --2- 172.21.15.102:0/2190266339 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f7d00039600 0x7f7d0003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d04006fd0 tx=0x7f7d04006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:23:58.472 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.536+0000 7f7d16000700 1 -- 172.21.15.102:0/2190266339 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7d0c00e260 con 0x7f7d1810af70 2024-04-26T20:23:58.472 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.762+0000 7f7d20200700 1 -- 172.21.15.102:0/2190266339 --> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f7d18108c10 con 0x7f7d00039600 2024-04-26T20:23:58.472 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.763+0000 7f7d16000700 1 -- 172.21.15.102:0/2190266339 <== mgr.14120 v2:172.21.15.102:6800/1441707916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f7d18108c10 con 0x7f7d00039600 2024-04-26T20:23:58.473 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.767+0000 7f7d20200700 1 -- 172.21.15.102:0/2190266339 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f7d00039600 msgr2=0x7f7d0003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:58.473 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.767+0000 7f7d20200700 1 --2- 172.21.15.102:0/2190266339 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f7d00039600 0x7f7d0003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d04006fd0 tx=0x7f7d04006e40 comp rx=0 tx=0).stop 2024-04-26T20:23:58.473 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.767+0000 7f7d20200700 1 -- 172.21.15.102:0/2190266339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 msgr2=0x7f7d1819ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:23:58.473 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.767+0000 7f7d20200700 1 --2- 172.21.15.102:0/2190266339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1819ef50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0117b0 tx=0x7f7d0c011b70 comp rx=0 tx=0).stop 2024-04-26T20:23:58.473 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.767+0000 7f7d20200700 1 -- 172.21.15.102:0/2190266339 shutdown_connections 2024-04-26T20:23:58.473 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.767+0000 7f7d20200700 1 --2- 172.21.15.102:0/2190266339 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f7d00039600 0x7f7d0003bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:58.473 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.767+0000 7f7d20200700 1 --2- 172.21.15.102:0/2190266339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d1810af70 0x7f7d1819ef50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:23:58.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.767+0000 7f7d20200700 1 -- 172.21.15.102:0/2190266339 >> 172.21.15.102:0/2190266339 conn(0x7f7d1807e370 msgr2=0x7f7d18108500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:23:58.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.768+0000 7f7d20200700 1 -- 172.21.15.102:0/2190266339 shutdown_connections 2024-04-26T20:23:58.474 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:57.768+0000 7f7d20200700 1 -- 172.21.15.102:0/2190266339 wait complete. 2024-04-26T20:23:58.474 INFO:teuthology.orchestra.run.smithi102.stdout:Generating ssh key... 2024-04-26T20:23:59.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:58 smithi102 ceph-mon[28094]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:24:00.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:59 smithi102 ceph-mon[28094]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:24:00.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:23:59 smithi102 ceph-mon[28094]: Generating ssh key... 2024-04-26T20:24:00.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: Generating public/private rsa key pair. 2024-04-26T20:24:00.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: Your identification has been saved in /tmp/tmpx5xaob7h/key. 2024-04-26T20:24:00.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: Your public key has been saved in /tmp/tmpx5xaob7h/key.pub. 2024-04-26T20:24:00.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: The key fingerprint is: 2024-04-26T20:24:00.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: SHA256:xpfSCl0mFIwE+79usfGyah5eQBxtnudtjZ1iZ9cdGNQ ceph-a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:24:00.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: The key's randomart image is: 2024-04-26T20:24:00.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: +---[RSA 3072]----+ 2024-04-26T20:24:00.400 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: | .oo+o. ... | 2024-04-26T20:24:00.400 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: | o.o+ . E | 2024-04-26T20:24:00.400 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: | . oo..o o | 2024-04-26T20:24:00.400 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: | o oo=.. . . | 2024-04-26T20:24:00.400 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: | + So+. + .+| 2024-04-26T20:24:00.400 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: | =o+. * * +| 2024-04-26T20:24:00.400 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: | . += o + . | 2024-04-26T20:24:00.400 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: | ..o+.. | 2024-04-26T20:24:00.400 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: | o+++o | 2024-04-26T20:24:00.400 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:00 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: +----[SHA256]-----+ 2024-04-26T20:24:00.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.084+0000 7f41f7200700 1 Processor -- start 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.084+0000 7f41f7200700 1 -- start start 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.085+0000 7f41f7200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f810b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.085+0000 7f41f7200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f41f810b7b0 con 0x7f41f810adc0 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.085+0000 7f41f5e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f810b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.085+0000 7f41f5e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f810b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36886/0 (socket says 172.21.15.102:36886) 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.085+0000 7f41f5e00700 1 -- 172.21.15.102:0/4076171619 learned_addr learned my addr 172.21.15.102:0/4076171619 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.086+0000 7f41f5e00700 1 -- 172.21.15.102:0/4076171619 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f810bfc0 con 0x7f41f810adc0 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.086+0000 7f41f5e00700 1 --2- 172.21.15.102:0/4076171619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f810b1e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f41e80098a0 tx=0x7f41e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91616bf6606ab96c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.086+0000 7f41f4a00700 1 -- 172.21.15.102:0/4076171619 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e8004160 con 0x7f41f810adc0 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.086+0000 7f41f4a00700 1 -- 172.21.15.102:0/4076171619 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f41e8004300 con 0x7f41f810adc0 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.086+0000 7f41f4a00700 1 -- 172.21.15.102:0/4076171619 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e8011af0 con 0x7f41f810adc0 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.087+0000 7f41f7200700 1 -- 172.21.15.102:0/4076171619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 msgr2=0x7f41f810b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:00.775 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.087+0000 7f41f7200700 1 --2- 172.21.15.102:0/4076171619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f810b1e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f41e80098a0 tx=0x7f41e8009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.087+0000 7f41f7200700 1 -- 172.21.15.102:0/4076171619 shutdown_connections 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.087+0000 7f41f7200700 1 --2- 172.21.15.102:0/4076171619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f810b1e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.087+0000 7f41f7200700 1 -- 172.21.15.102:0/4076171619 >> 172.21.15.102:0/4076171619 conn(0x7f41f807e190 msgr2=0x7f41f807e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.087+0000 7f41f7200700 1 -- 172.21.15.102:0/4076171619 shutdown_connections 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.087+0000 7f41f7200700 1 -- 172.21.15.102:0/4076171619 wait complete. 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.088+0000 7f41f7200700 1 Processor -- start 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.088+0000 7f41f7200700 1 -- start start 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.089+0000 7f41f7200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.089+0000 7f41f7200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f41f810f370 con 0x7f41f810adc0 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.089+0000 7f41f5e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.089+0000 7f41f5e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f819eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36888/0 (socket says 172.21.15.102:36888) 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.089+0000 7f41f5e00700 1 -- 172.21.15.102:0/2641252981 learned_addr learned my addr 172.21.15.102:0/2641252981 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.089+0000 7f41f5e00700 1 -- 172.21.15.102:0/2641252981 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f819f2e0 con 0x7f41f810adc0 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.090+0000 7f41f5e00700 1 --2- 172.21.15.102:0/2641252981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f819eda0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f41e80117b0 tx=0x7f41e8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:00.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.090+0000 7f41eea00700 1 -- 172.21.15.102:0/2641252981 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e8003af0 con 0x7f41f810adc0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.090+0000 7f41eea00700 1 -- 172.21.15.102:0/2641252981 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f41e8020070 con 0x7f41f810adc0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.090+0000 7f41eea00700 1 -- 172.21.15.102:0/2641252981 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41e8019630 con 0x7f41f810adc0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.090+0000 7f41f7200700 1 -- 172.21.15.102:0/2641252981 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f819f570 con 0x7f41f810adc0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.090+0000 7f41f7200700 1 -- 172.21.15.102:0/2641252981 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f81a2260 con 0x7f41f810adc0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.091+0000 7f41eea00700 1 -- 172.21.15.102:0/2641252981 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f41e801a9c0 con 0x7f41f810adc0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.091+0000 7f41f7200700 1 -- 172.21.15.102:0/2641252981 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f806a540 con 0x7f41f810adc0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.092+0000 7f41eea00700 1 --2- 172.21.15.102:0/2641252981 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f41dc039600 0x7f41dc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.092+0000 7f41eea00700 1 -- 172.21.15.102:0/2641252981 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f41e804dbf0 con 0x7f41f810adc0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.092+0000 7f41f5400700 1 --2- 172.21.15.102:0/2641252981 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f41dc039600 0x7f41dc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.093+0000 7f41f5400700 1 --2- 172.21.15.102:0/2641252981 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f41dc039600 0x7f41dc03bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f41e4006fd0 tx=0x7f41e4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.098+0000 7f41eea00700 1 -- 172.21.15.102:0/2641252981 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f41e80042a0 con 0x7f41f810adc0 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:23:59.326+0000 7f41f7200700 1 -- 172.21.15.102:0/2641252981 --> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f41f8108c40 con 0x7f41dc039600 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.144+0000 7f41eea00700 1 -- 172.21.15.102:0/2641252981 <== mgr.14120 v2:172.21.15.102:6800/1441707916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f41f8108c40 con 0x7f41dc039600 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.146+0000 7f41f7200700 1 -- 172.21.15.102:0/2641252981 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f41dc039600 msgr2=0x7f41dc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:00.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.146+0000 7f41f7200700 1 --2- 172.21.15.102:0/2641252981 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f41dc039600 0x7f41dc03bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f41e4006fd0 tx=0x7f41e4006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:00.778 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.146+0000 7f41f7200700 1 -- 172.21.15.102:0/2641252981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 msgr2=0x7f41f819eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:00.778 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.146+0000 7f41f7200700 1 --2- 172.21.15.102:0/2641252981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f819eda0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f41e80117b0 tx=0x7f41e8011b70 comp rx=0 tx=0).stop 2024-04-26T20:24:00.778 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.147+0000 7f41f7200700 1 -- 172.21.15.102:0/2641252981 shutdown_connections 2024-04-26T20:24:00.778 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.147+0000 7f41f7200700 1 --2- 172.21.15.102:0/2641252981 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f41dc039600 0x7f41dc03bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:00.778 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.147+0000 7f41f7200700 1 --2- 172.21.15.102:0/2641252981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41f810adc0 0x7f41f819eda0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:00.778 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.147+0000 7f41f7200700 1 -- 172.21.15.102:0/2641252981 >> 172.21.15.102:0/2641252981 conn(0x7f41f807e190 msgr2=0x7f41f8108530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:00.778 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.147+0000 7f41f7200700 1 -- 172.21.15.102:0/2641252981 shutdown_connections 2024-04-26T20:24:00.778 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:00.147+0000 7f41f7200700 1 -- 172.21.15.102:0/2641252981 wait complete. 2024-04-26T20:24:01.392 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:01 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:24:01.392 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:01 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:24:02.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:02 smithi102 ceph-mon[28094]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:24:02.420 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfx7g9Ew1nCB0wLtv6SVsQ621pqMc/t/JIRUPBeX7P/aZK0xLM2ec8mVL/qn4XekcfMMaPNqhprCI8C9zgevKYoYRWVds+UftjnKYOqpy89WXVw2n9LSWlJYvVr8A7FhvcHcelSlnTaeMkuli31C/DH+y0ZqdFxm81uJo/0nCw6ZZulhjMbkZDjWzq5IFiDUQ7uSWo428HzWMCKgt+fM4IhGCWZVbRXSaexo8CeMMqadrrQISsHBnIEQv2ni9sNyKHQhD8L6iVBAtq7o1SnlomupaY9G098jRYrE3YLQ6xtqKG6lTR7NHUEQAOuc5IsqlRfnP70cFlb/gnoYkkJzmgkiCUvz2uK82d/YiZKLhT+NII7++0QbNsPf6A64Dm8kTzvICec3Ws9DQ1bn+2KOniieLbPAsL3ZS2wWIoRRbaxCHE6VNeqkrntxRkIsIZ01NrLtutAB/4W9bbCvU1RPwcO4WGnELGlKCiszqfeKJGodxJD6fu8/ZHwzPws0SPKr0= ceph-a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:24:02.421 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.390+0000 7f8e91c00700 1 Processor -- start 2024-04-26T20:24:02.421 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.390+0000 7f8e91c00700 1 -- start start 2024-04-26T20:24:02.422 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.390+0000 7f8e91c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c10b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:02.422 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.390+0000 7f8e91c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c10b7b0 con 0x7f8e8c10adc0 2024-04-26T20:24:02.422 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.391+0000 7f8e8be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c10b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:02.422 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.391+0000 7f8e8be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c10b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36900/0 (socket says 172.21.15.102:36900) 2024-04-26T20:24:02.423 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.391+0000 7f8e8be00700 1 -- 172.21.15.102:0/3256427563 learned_addr learned my addr 172.21.15.102:0/3256427563 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:02.423 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.392+0000 7f8e8be00700 1 -- 172.21.15.102:0/3256427563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8c10bf70 con 0x7f8e8c10adc0 2024-04-26T20:24:02.423 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.392+0000 7f8e8be00700 1 --2- 172.21.15.102:0/3256427563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c10b1e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0098a0 tx=0x7f8e7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5600fd90de0bddbb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:02.423 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.393+0000 7f8e8aa00700 1 -- 172.21.15.102:0/3256427563 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e7c004160 con 0x7f8e8c10adc0 2024-04-26T20:24:02.423 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.393+0000 7f8e8aa00700 1 -- 172.21.15.102:0/3256427563 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8e7c004300 con 0x7f8e8c10adc0 2024-04-26T20:24:02.423 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.393+0000 7f8e8aa00700 1 -- 172.21.15.102:0/3256427563 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e7c011af0 con 0x7f8e8c10adc0 2024-04-26T20:24:02.423 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.393+0000 7f8e91c00700 1 -- 172.21.15.102:0/3256427563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 msgr2=0x7f8e8c10b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:02.425 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.393+0000 7f8e91c00700 1 --2- 172.21.15.102:0/3256427563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c10b1e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0098a0 tx=0x7f8e7c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:02.425 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.393+0000 7f8e91c00700 1 -- 172.21.15.102:0/3256427563 shutdown_connections 2024-04-26T20:24:02.425 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.393+0000 7f8e91c00700 1 --2- 172.21.15.102:0/3256427563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c10b1e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:02.425 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.393+0000 7f8e91c00700 1 -- 172.21.15.102:0/3256427563 >> 172.21.15.102:0/3256427563 conn(0x7f8e8c07e1f0 msgr2=0x7f8e8c07e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:02.425 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.393+0000 7f8e91c00700 1 -- 172.21.15.102:0/3256427563 shutdown_connections 2024-04-26T20:24:02.425 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.393+0000 7f8e91c00700 1 -- 172.21.15.102:0/3256427563 wait complete. 2024-04-26T20:24:02.425 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.394+0000 7f8e91c00700 1 Processor -- start 2024-04-26T20:24:02.425 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.395+0000 7f8e91c00700 1 -- start start 2024-04-26T20:24:02.425 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.395+0000 7f8e91c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c19ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:02.425 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.395+0000 7f8e91c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c10f320 con 0x7f8e8c10adc0 2024-04-26T20:24:02.426 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.395+0000 7f8e8be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c19ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:02.426 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.395+0000 7f8e8be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c19ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36916/0 (socket says 172.21.15.102:36916) 2024-04-26T20:24:02.428 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.395+0000 7f8e8be00700 1 -- 172.21.15.102:0/4033270645 learned_addr learned my addr 172.21.15.102:0/4033270645 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:02.428 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.395+0000 7f8e8be00700 1 -- 172.21.15.102:0/4033270645 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8c19f220 con 0x7f8e8c10adc0 2024-04-26T20:24:02.428 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.396+0000 7f8e8be00700 1 --2- 172.21.15.102:0/4033270645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c19ece0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c003f40 tx=0x7f8e7c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:02.428 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.396+0000 7f8e88c00700 1 -- 172.21.15.102:0/4033270645 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e7c0038b0 con 0x7f8e8c10adc0 2024-04-26T20:24:02.428 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.396+0000 7f8e88c00700 1 -- 172.21.15.102:0/4033270645 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8e7c003a50 con 0x7f8e8c10adc0 2024-04-26T20:24:02.428 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.396+0000 7f8e91c00700 1 -- 172.21.15.102:0/4033270645 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8c19f4b0 con 0x7f8e8c10adc0 2024-04-26T20:24:02.428 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.396+0000 7f8e88c00700 1 -- 172.21.15.102:0/4033270645 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e7c01a5c0 con 0x7f8e8c10adc0 2024-04-26T20:24:02.428 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.396+0000 7f8e91c00700 1 -- 172.21.15.102:0/4033270645 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e8c1a2190 con 0x7f8e8c10adc0 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.397+0000 7f8e88c00700 1 -- 172.21.15.102:0/4033270645 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f8e7c024430 con 0x7f8e8c10adc0 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.397+0000 7f8e91c00700 1 -- 172.21.15.102:0/4033270645 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e8c06a540 con 0x7f8e8c10adc0 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.398+0000 7f8e88c00700 1 --2- 172.21.15.102:0/4033270645 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f8e700395b0 0x7f8e7003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.398+0000 7f8e88c00700 1 -- 172.21.15.102:0/4033270645 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8e7c04db50 con 0x7f8e8c10adc0 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.398+0000 7f8e8b400700 1 --2- 172.21.15.102:0/4033270645 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f8e700395b0 0x7f8e7003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.399+0000 7f8e8b400700 1 --2- 172.21.15.102:0/4033270645 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f8e700395b0 0x7f8e7003ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8e78006fd0 tx=0x7f8e78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.403+0000 7f8e88c00700 1 -- 172.21.15.102:0/4033270645 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8e7c04d4a0 con 0x7f8e8c10adc0 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.630+0000 7f8e91c00700 1 -- 172.21.15.102:0/4033270645 --> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f8e8c108be0 con 0x7f8e700395b0 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.632+0000 7f8e88c00700 1 -- 172.21.15.102:0/4033270645 <== mgr.14120 v2:172.21.15.102:6800/1441707916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f8e8c108be0 con 0x7f8e700395b0 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.635+0000 7f8e91c00700 1 -- 172.21.15.102:0/4033270645 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f8e700395b0 msgr2=0x7f8e7003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.635+0000 7f8e91c00700 1 --2- 172.21.15.102:0/4033270645 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f8e700395b0 0x7f8e7003ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8e78006fd0 tx=0x7f8e78006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.635+0000 7f8e91c00700 1 -- 172.21.15.102:0/4033270645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 msgr2=0x7f8e8c19ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:02.429 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.635+0000 7f8e91c00700 1 --2- 172.21.15.102:0/4033270645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c19ece0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c003f40 tx=0x7f8e7c011890 comp rx=0 tx=0).stop 2024-04-26T20:24:02.430 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.635+0000 7f8e91c00700 1 -- 172.21.15.102:0/4033270645 shutdown_connections 2024-04-26T20:24:02.431 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.635+0000 7f8e91c00700 1 --2- 172.21.15.102:0/4033270645 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f8e700395b0 0x7f8e7003ba70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:02.431 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.635+0000 7f8e91c00700 1 --2- 172.21.15.102:0/4033270645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e8c10adc0 0x7f8e8c19ece0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:02.431 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.635+0000 7f8e91c00700 1 -- 172.21.15.102:0/4033270645 >> 172.21.15.102:0/4033270645 conn(0x7f8e8c07e1f0 msgr2=0x7f8e8c1084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:02.431 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.635+0000 7f8e91c00700 1 -- 172.21.15.102:0/4033270645 shutdown_connections 2024-04-26T20:24:02.431 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:01.635+0000 7f8e91c00700 1 -- 172.21.15.102:0/4033270645 wait complete. 2024-04-26T20:24:02.431 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T20:24:02.431 INFO:teuthology.orchestra.run.smithi102.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T20:24:02.441 INFO:teuthology.orchestra.run.smithi102.stdout:Adding host smithi102... 2024-04-26T20:24:04.260 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:03 smithi102 ceph-mon[28094]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:24:05.119 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:04 smithi102 ceph-mon[28094]: Deploying cephadm binary to smithi102 2024-04-26T20:24:10.100 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Added host 'smithi102' with addr '172.21.15.102' 2024-04-26T20:24:10.100 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.074+0000 7fef97e00700 1 Processor -- start 2024-04-26T20:24:10.100 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.074+0000 7fef97e00700 1 -- start start 2024-04-26T20:24:10.100 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.075+0000 7fef97e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef98108fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:10.100 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.075+0000 7fef97e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fef98109580 con 0x7fef98108b90 2024-04-26T20:24:10.100 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.075+0000 7fef96a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef98108fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.075+0000 7fef96a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef98108fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36924/0 (socket says 172.21.15.102:36924) 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.075+0000 7fef96a00700 1 -- 172.21.15.102:0/4274513563 learned_addr learned my addr 172.21.15.102:0/4274513563 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.076+0000 7fef96a00700 1 -- 172.21.15.102:0/4274513563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef98109d40 con 0x7fef98108b90 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.076+0000 7fef96a00700 1 --2- 172.21.15.102:0/4274513563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef98108fb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fef880098a0 tx=0x7fef88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2997f702c9545c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.076+0000 7fef96000700 1 -- 172.21.15.102:0/4274513563 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef88004160 con 0x7fef98108b90 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.076+0000 7fef96000700 1 -- 172.21.15.102:0/4274513563 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fef88004300 con 0x7fef98108b90 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.076+0000 7fef96000700 1 -- 172.21.15.102:0/4274513563 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef88011af0 con 0x7fef98108b90 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.077+0000 7fef97e00700 1 -- 172.21.15.102:0/4274513563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 msgr2=0x7fef98108fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.077+0000 7fef97e00700 1 --2- 172.21.15.102:0/4274513563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef98108fb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fef880098a0 tx=0x7fef88009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.077+0000 7fef97e00700 1 -- 172.21.15.102:0/4274513563 shutdown_connections 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.077+0000 7fef97e00700 1 --2- 172.21.15.102:0/4274513563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef98108fb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:10.101 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.077+0000 7fef97e00700 1 -- 172.21.15.102:0/4274513563 >> 172.21.15.102:0/4274513563 conn(0x7fef981040f0 msgr2=0x7fef98106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.077+0000 7fef97e00700 1 -- 172.21.15.102:0/4274513563 shutdown_connections 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.077+0000 7fef97e00700 1 -- 172.21.15.102:0/4274513563 wait complete. 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.078+0000 7fef97e00700 1 Processor -- start 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.079+0000 7fef97e00700 1 -- start start 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.079+0000 7fef97e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef9819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.079+0000 7fef97e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fef9810d0f0 con 0x7fef98108b90 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.079+0000 7fef96a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef9819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.079+0000 7fef96a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef9819ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36940/0 (socket says 172.21.15.102:36940) 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.079+0000 7fef96a00700 1 -- 172.21.15.102:0/1473165019 learned_addr learned my addr 172.21.15.102:0/1473165019 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.080+0000 7fef96a00700 1 -- 172.21.15.102:0/1473165019 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9819f320 con 0x7fef98108b90 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.080+0000 7fef96a00700 1 --2- 172.21.15.102:0/1473165019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef9819ede0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fef880117b0 tx=0x7fef88011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.080+0000 7fef9ca00700 1 -- 172.21.15.102:0/1473165019 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef88003af0 con 0x7fef98108b90 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.080+0000 7fef9ca00700 1 -- 172.21.15.102:0/1473165019 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fef88020070 con 0x7fef98108b90 2024-04-26T20:24:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.080+0000 7fef9ca00700 1 -- 172.21.15.102:0/1473165019 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef88019630 con 0x7fef98108b90 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.080+0000 7fef97e00700 1 -- 172.21.15.102:0/1473165019 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef9819f5b0 con 0x7fef98108b90 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.080+0000 7fef97e00700 1 -- 172.21.15.102:0/1473165019 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef981a22a0 con 0x7fef98108b90 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.081+0000 7fef97e00700 1 -- 172.21.15.102:0/1473165019 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef9806a540 con 0x7fef98108b90 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.081+0000 7fef9ca00700 1 -- 172.21.15.102:0/1473165019 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fef8801a9c0 con 0x7fef98108b90 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.082+0000 7fef9ca00700 1 --2- 172.21.15.102:0/1473165019 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fef7c039660 0x7fef7c03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.082+0000 7fef9ca00700 1 -- 172.21.15.102:0/1473165019 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fef8804dbb0 con 0x7fef98108b90 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.082+0000 7fef8e000700 1 --2- 172.21.15.102:0/1473165019 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fef7c039660 0x7fef7c03bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.083+0000 7fef8e000700 1 --2- 172.21.15.102:0/1473165019 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fef7c039660 0x7fef7c03bb20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fef80006fd0 tx=0x7fef80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.088+0000 7fef9ca00700 1 -- 172.21.15.102:0/1473165019 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fef8800e260 con 0x7fef98108b90 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:03.318+0000 7fef97e00700 1 -- 172.21.15.102:0/1473165019 --> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}) -- 0x7fef98105410 con 0x7fef7c039660 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.363+0000 7fef9ca00700 1 -- 172.21.15.102:0/1473165019 <== mgr.14120 v2:172.21.15.102:6800/1441707916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fef98105410 con 0x7fef7c039660 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.367+0000 7fef97e00700 1 -- 172.21.15.102:0/1473165019 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fef7c039660 msgr2=0x7fef7c03bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:10.103 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.367+0000 7fef97e00700 1 --2- 172.21.15.102:0/1473165019 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fef7c039660 0x7fef7c03bb20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fef80006fd0 tx=0x7fef80006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:10.104 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.367+0000 7fef97e00700 1 -- 172.21.15.102:0/1473165019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 msgr2=0x7fef9819ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:10.104 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.367+0000 7fef97e00700 1 --2- 172.21.15.102:0/1473165019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef9819ede0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fef880117b0 tx=0x7fef88011b70 comp rx=0 tx=0).stop 2024-04-26T20:24:10.104 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.367+0000 7fef97e00700 1 -- 172.21.15.102:0/1473165019 shutdown_connections 2024-04-26T20:24:10.104 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.367+0000 7fef97e00700 1 --2- 172.21.15.102:0/1473165019 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fef7c039660 0x7fef7c03bb20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:10.104 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.367+0000 7fef97e00700 1 --2- 172.21.15.102:0/1473165019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef98108b90 0x7fef9819ede0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:10.104 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.367+0000 7fef97e00700 1 -- 172.21.15.102:0/1473165019 >> 172.21.15.102:0/1473165019 conn(0x7fef981040f0 msgr2=0x7fef98104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:10.104 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.368+0000 7fef97e00700 1 -- 172.21.15.102:0/1473165019 shutdown_connections 2024-04-26T20:24:10.104 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:09.368+0000 7fef97e00700 1 -- 172.21.15.102:0/1473165019 wait complete. 2024-04-26T20:24:10.104 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mon service... 2024-04-26T20:24:10.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:10 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:24:10.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:10 smithi102 ceph-mon[28094]: Added host smithi102 2024-04-26T20:24:10.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:10 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:12.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:12 smithi102 ceph-mon[28094]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:24:12.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:12 smithi102 ceph-mon[28094]: Saving service mon spec with placement count:5 2024-04-26T20:24:12.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:12 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:24:12.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T20:24:12.541 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.777+0000 7f3ede800700 1 Processor -- start 2024-04-26T20:24:12.542 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.777+0000 7f3ede800700 1 -- start start 2024-04-26T20:24:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.778+0000 7f3ede800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed807d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.778+0000 7f3ede800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed807d8c0 con 0x7f3ed807ee90 2024-04-26T20:24:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.778+0000 7f3ed7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed807d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.778+0000 7f3ed7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed807d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37522/0 (socket says 172.21.15.102:37522) 2024-04-26T20:24:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.778+0000 7f3ed7400700 1 -- 172.21.15.102:0/1371189473 learned_addr learned my addr 172.21.15.102:0/1371189473 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.779+0000 7f3ed7400700 1 -- 172.21.15.102:0/1371189473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed807da40 con 0x7f3ed807ee90 2024-04-26T20:24:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.779+0000 7f3ed7400700 1 --2- 172.21.15.102:0/1371189473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed807d2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80098a0 tx=0x7f3ec8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7c0cf6c080f3f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.779+0000 7f3ed6a00700 1 -- 172.21.15.102:0/1371189473 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ec8004160 con 0x7f3ed807ee90 2024-04-26T20:24:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.779+0000 7f3ed6a00700 1 -- 172.21.15.102:0/1371189473 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3ec8004300 con 0x7f3ed807ee90 2024-04-26T20:24:12.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.779+0000 7f3ed6a00700 1 -- 172.21.15.102:0/1371189473 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ec8011af0 con 0x7f3ed807ee90 2024-04-26T20:24:12.545 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.780+0000 7f3ede800700 1 -- 172.21.15.102:0/1371189473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 msgr2=0x7f3ed807d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.780+0000 7f3ede800700 1 --2- 172.21.15.102:0/1371189473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed807d2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80098a0 tx=0x7f3ec8009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.780+0000 7f3ede800700 1 -- 172.21.15.102:0/1371189473 shutdown_connections 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.780+0000 7f3ede800700 1 --2- 172.21.15.102:0/1371189473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed807d2f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.780+0000 7f3ede800700 1 -- 172.21.15.102:0/1371189473 >> 172.21.15.102:0/1371189473 conn(0x7f3ed8104370 msgr2=0x7f3ed8106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.780+0000 7f3ede800700 1 -- 172.21.15.102:0/1371189473 shutdown_connections 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.780+0000 7f3ede800700 1 -- 172.21.15.102:0/1371189473 wait complete. 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.781+0000 7f3ede800700 1 Processor -- start 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.781+0000 7f3ede800700 1 -- start start 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.782+0000 7f3ede800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.782+0000 7f3ede800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed810f320 con 0x7f3ed807ee90 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.782+0000 7f3ed7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:12.546 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.782+0000 7f3ed7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed819ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37534/0 (socket says 172.21.15.102:37534) 2024-04-26T20:24:12.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.782+0000 7f3ed7400700 1 -- 172.21.15.102:0/2630547802 learned_addr learned my addr 172.21.15.102:0/2630547802 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:12.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.782+0000 7f3ed7400700 1 -- 172.21.15.102:0/2630547802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed819f470 con 0x7f3ed807ee90 2024-04-26T20:24:12.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.783+0000 7f3ed7400700 1 --2- 172.21.15.102:0/2630547802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed819ef30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8000c00 tx=0x7f3ec8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:12.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.783+0000 7f3ed4c00700 1 -- 172.21.15.102:0/2630547802 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ec8003b20 con 0x7f3ed807ee90 2024-04-26T20:24:12.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.783+0000 7f3ed4c00700 1 -- 172.21.15.102:0/2630547802 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3ec801a8c0 con 0x7f3ed807ee90 2024-04-26T20:24:12.547 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.783+0000 7f3ed4c00700 1 -- 172.21.15.102:0/2630547802 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ec8024940 con 0x7f3ed807ee90 2024-04-26T20:24:12.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.783+0000 7f3ede800700 1 -- 172.21.15.102:0/2630547802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed819f700 con 0x7f3ed807ee90 2024-04-26T20:24:12.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.783+0000 7f3ede800700 1 -- 172.21.15.102:0/2630547802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed81a23f0 con 0x7f3ed807ee90 2024-04-26T20:24:12.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.784+0000 7f3ede800700 1 -- 172.21.15.102:0/2630547802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ed8064a80 con 0x7f3ed807ee90 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.785+0000 7f3ed4c00700 1 -- 172.21.15.102:0/2630547802 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3ec8022020 con 0x7f3ed807ee90 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.785+0000 7f3ed4c00700 1 --2- 172.21.15.102:0/2630547802 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f3ebc039600 0x7f3ebc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.785+0000 7f3ed4c00700 1 -- 172.21.15.102:0/2630547802 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3ec804d7e0 con 0x7f3ed807ee90 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.785+0000 7f3ecea00700 1 --2- 172.21.15.102:0/2630547802 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f3ebc039600 0x7f3ebc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.786+0000 7f3ecea00700 1 --2- 172.21.15.102:0/2630547802 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f3ebc039600 0x7f3ebc03bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0006fd0 tx=0x7f3ec0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:10.791+0000 7f3ed4c00700 1 -- 172.21.15.102:0/2630547802 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3ec800be50 con 0x7f3ed807ee90 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.036+0000 7f3ede800700 1 -- 172.21.15.102:0/2630547802 --> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f3ed81055c0 con 0x7f3ebc039600 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.043+0000 7f3ed4c00700 1 -- 172.21.15.102:0/2630547802 <== mgr.14120 v2:172.21.15.102:6800/1441707916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3ed81055c0 con 0x7f3ebc039600 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.046+0000 7f3ede800700 1 -- 172.21.15.102:0/2630547802 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f3ebc039600 msgr2=0x7f3ebc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.046+0000 7f3ede800700 1 --2- 172.21.15.102:0/2630547802 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f3ebc039600 0x7f3ebc03bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0006fd0 tx=0x7f3ec0006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.046+0000 7f3ede800700 1 -- 172.21.15.102:0/2630547802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 msgr2=0x7f3ed819ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.046+0000 7f3ede800700 1 --2- 172.21.15.102:0/2630547802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed819ef30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8000c00 tx=0x7f3ec8011b70 comp rx=0 tx=0).stop 2024-04-26T20:24:12.550 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.047+0000 7f3ede800700 1 -- 172.21.15.102:0/2630547802 shutdown_connections 2024-04-26T20:24:12.551 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.047+0000 7f3ede800700 1 --2- 172.21.15.102:0/2630547802 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f3ebc039600 0x7f3ebc03bac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:12.551 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.047+0000 7f3ede800700 1 --2- 172.21.15.102:0/2630547802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed807ee90 0x7f3ed819ef30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:12.551 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.047+0000 7f3ede800700 1 -- 172.21.15.102:0/2630547802 >> 172.21.15.102:0/2630547802 conn(0x7f3ed8104370 msgr2=0x7f3ed8104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:12.551 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.047+0000 7f3ede800700 1 -- 172.21.15.102:0/2630547802 shutdown_connections 2024-04-26T20:24:12.551 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:11.047+0000 7f3ede800700 1 -- 172.21.15.102:0/2630547802 wait complete. 2024-04-26T20:24:12.551 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mgr service... 2024-04-26T20:24:14.319 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:13 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:24:14.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:13 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:24:14.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:13 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:24:14.332 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T20:24:14.332 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.393+0000 7f48fe000700 1 Processor -- start 2024-04-26T20:24:14.332 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.393+0000 7f48fe000700 1 -- start start 2024-04-26T20:24:14.332 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.394+0000 7f48fe000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f810bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:14.332 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.394+0000 7f48fe000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f48f807ce60 con 0x7f48f8109870 2024-04-26T20:24:14.332 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.394+0000 7f48f7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f810bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:14.332 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.394+0000 7f48f7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f810bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37548/0 (socket says 172.21.15.102:37548) 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.394+0000 7f48f7400700 1 -- 172.21.15.102:0/1703479357 learned_addr learned my addr 172.21.15.102:0/1703479357 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.395+0000 7f48f7400700 1 -- 172.21.15.102:0/1703479357 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f810c1a0 con 0x7f48f8109870 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.395+0000 7f48f7400700 1 --2- 172.21.15.102:0/1703479357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f810bc60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f48e80098a0 tx=0x7f48e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdfe41e8a73c6a7e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.395+0000 7f48f6000700 1 -- 172.21.15.102:0/1703479357 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48e8004160 con 0x7f48f8109870 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.395+0000 7f48f6000700 1 -- 172.21.15.102:0/1703479357 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48e8004300 con 0x7f48f8109870 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.395+0000 7f48f6000700 1 -- 172.21.15.102:0/1703479357 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48e8011a20 con 0x7f48f8109870 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.396+0000 7f48fe000700 1 -- 172.21.15.102:0/1703479357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 msgr2=0x7f48f810bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.396+0000 7f48fe000700 1 --2- 172.21.15.102:0/1703479357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f810bc60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f48e80098a0 tx=0x7f48e8009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.396+0000 7f48fe000700 1 -- 172.21.15.102:0/1703479357 shutdown_connections 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.396+0000 7f48fe000700 1 --2- 172.21.15.102:0/1703479357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f810bc60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.396+0000 7f48fe000700 1 -- 172.21.15.102:0/1703479357 >> 172.21.15.102:0/1703479357 conn(0x7f48f8103260 msgr2=0x7f48f81056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:14.333 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.396+0000 7f48fe000700 1 -- 172.21.15.102:0/1703479357 shutdown_connections 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.396+0000 7f48fe000700 1 -- 172.21.15.102:0/1703479357 wait complete. 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.397+0000 7f48fe000700 1 Processor -- start 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.398+0000 7f48fe000700 1 -- start start 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.398+0000 7f48fe000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f81090f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.398+0000 7f48fe000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f48f810d1a0 con 0x7f48f8109870 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.398+0000 7f48f7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f81090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.398+0000 7f48f7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f81090f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37556/0 (socket says 172.21.15.102:37556) 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.398+0000 7f48f7400700 1 -- 172.21.15.102:0/3956781162 learned_addr learned my addr 172.21.15.102:0/3956781162 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.399+0000 7f48f7400700 1 -- 172.21.15.102:0/3956781162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f8109630 con 0x7f48f8109870 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.399+0000 7f48f7400700 1 --2- 172.21.15.102:0/3956781162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f81090f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f48e8004660 tx=0x7f48e8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.399+0000 7f48efe00700 1 -- 172.21.15.102:0/3956781162 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48e8003aa0 con 0x7f48f8109870 2024-04-26T20:24:14.334 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.399+0000 7f48efe00700 1 -- 172.21.15.102:0/3956781162 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48e801b8f0 con 0x7f48f8109870 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.399+0000 7f48efe00700 1 -- 172.21.15.102:0/3956781162 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48e8023900 con 0x7f48f8109870 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.399+0000 7f48fe000700 1 -- 172.21.15.102:0/3956781162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48f8107820 con 0x7f48f8109870 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.400+0000 7f48fe000700 1 -- 172.21.15.102:0/3956781162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48f8107d60 con 0x7f48f8109870 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.401+0000 7f48fe000700 1 -- 172.21.15.102:0/3956781162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48f8064a80 con 0x7f48f8109870 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.401+0000 7f48efe00700 1 -- 172.21.15.102:0/3956781162 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f48e801baa0 con 0x7f48f8109870 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.402+0000 7f48efe00700 1 --2- 172.21.15.102:0/3956781162 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f48dc039600 0x7f48dc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.402+0000 7f48efe00700 1 -- 172.21.15.102:0/3956781162 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f48e804df80 con 0x7f48f8109870 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.402+0000 7f48f6a00700 1 --2- 172.21.15.102:0/3956781162 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f48dc039600 0x7f48dc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.403+0000 7f48f6a00700 1 --2- 172.21.15.102:0/3956781162 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f48dc039600 0x7f48dc03bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f48e4006fd0 tx=0x7f48e4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.408+0000 7f48efe00700 1 -- 172.21.15.102:0/3956781162 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f48e801f030 con 0x7f48f8109870 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.653+0000 7f48fe000700 1 -- 172.21.15.102:0/3956781162 --> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f48f8108570 con 0x7f48dc039600 2024-04-26T20:24:14.335 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.660+0000 7f48efe00700 1 -- 172.21.15.102:0/3956781162 <== mgr.14120 v2:172.21.15.102:6800/1441707916 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f48f8108570 con 0x7f48dc039600 2024-04-26T20:24:14.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.663+0000 7f48fe000700 1 -- 172.21.15.102:0/3956781162 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f48dc039600 msgr2=0x7f48dc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:14.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.663+0000 7f48fe000700 1 --2- 172.21.15.102:0/3956781162 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f48dc039600 0x7f48dc03bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f48e4006fd0 tx=0x7f48e4006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:14.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.663+0000 7f48fe000700 1 -- 172.21.15.102:0/3956781162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 msgr2=0x7f48f81090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:14.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.663+0000 7f48fe000700 1 --2- 172.21.15.102:0/3956781162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f81090f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f48e8004660 tx=0x7f48e8011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:14.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.663+0000 7f48fe000700 1 -- 172.21.15.102:0/3956781162 shutdown_connections 2024-04-26T20:24:14.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.663+0000 7f48fe000700 1 --2- 172.21.15.102:0/3956781162 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f48dc039600 0x7f48dc03bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:14.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.663+0000 7f48fe000700 1 --2- 172.21.15.102:0/3956781162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48f8109870 0x7f48f81090f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:14.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.663+0000 7f48fe000700 1 -- 172.21.15.102:0/3956781162 >> 172.21.15.102:0/3956781162 conn(0x7f48f8103260 msgr2=0x7f48f8103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:14.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.663+0000 7f48fe000700 1 -- 172.21.15.102:0/3956781162 shutdown_connections 2024-04-26T20:24:14.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:13.663+0000 7f48fe000700 1 -- 172.21.15.102:0/3956781162 wait complete. 2024-04-26T20:24:15.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:14 smithi102 ceph-mon[28094]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:24:15.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:14 smithi102 ceph-mon[28094]: Saving service mgr spec with placement count:2 2024-04-26T20:24:15.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:14 smithi102 ceph-mon[28094]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:24:16.281 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.909+0000 7f1ca3600700 1 Processor -- start 2024-04-26T20:24:16.281 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.910+0000 7f1ca3600700 1 -- start start 2024-04-26T20:24:16.281 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.910+0000 7f1ca3600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c07d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:16.281 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.910+0000 7f1ca3600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c07d8a0 con 0x7f1c9c07ee70 2024-04-26T20:24:16.281 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.910+0000 7f1ca0c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c07d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:16.282 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.910+0000 7f1ca0c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c07d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37592/0 (socket says 172.21.15.102:37592) 2024-04-26T20:24:16.282 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.911+0000 7f1ca0c00700 1 -- 172.21.15.102:0/1903137468 learned_addr learned my addr 172.21.15.102:0/1903137468 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:16.282 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.911+0000 7f1ca0c00700 1 -- 172.21.15.102:0/1903137468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c07da20 con 0x7f1c9c07ee70 2024-04-26T20:24:16.282 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.911+0000 7f1ca0c00700 1 --2- 172.21.15.102:0/1903137468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c07d2d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c900098a0 tx=0x7f1c90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d20129c94cd80fa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:16.283 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.912+0000 7f1c9b400700 1 -- 172.21.15.102:0/1903137468 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c90004160 con 0x7f1c9c07ee70 2024-04-26T20:24:16.283 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.912+0000 7f1c9b400700 1 -- 172.21.15.102:0/1903137468 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c90004300 con 0x7f1c9c07ee70 2024-04-26T20:24:16.283 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.912+0000 7f1c9b400700 1 -- 172.21.15.102:0/1903137468 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c90011a20 con 0x7f1c9c07ee70 2024-04-26T20:24:16.283 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.912+0000 7f1ca3600700 1 -- 172.21.15.102:0/1903137468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 msgr2=0x7f1c9c07d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:16.283 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.912+0000 7f1ca3600700 1 --2- 172.21.15.102:0/1903137468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c07d2d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c900098a0 tx=0x7f1c90009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:16.284 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.913+0000 7f1ca3600700 1 -- 172.21.15.102:0/1903137468 shutdown_connections 2024-04-26T20:24:16.284 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.913+0000 7f1ca3600700 1 --2- 172.21.15.102:0/1903137468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c07d2d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:16.284 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.913+0000 7f1ca3600700 1 -- 172.21.15.102:0/1903137468 >> 172.21.15.102:0/1903137468 conn(0x7f1c9c104370 msgr2=0x7f1c9c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:16.284 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.913+0000 7f1ca3600700 1 -- 172.21.15.102:0/1903137468 shutdown_connections 2024-04-26T20:24:16.285 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.913+0000 7f1ca3600700 1 -- 172.21.15.102:0/1903137468 wait complete. 2024-04-26T20:24:16.285 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.914+0000 7f1ca3600700 1 Processor -- start 2024-04-26T20:24:16.285 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.914+0000 7f1ca3600700 1 -- start start 2024-04-26T20:24:16.285 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.914+0000 7f1ca3600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:16.285 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.914+0000 7f1ca3600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10f390 con 0x7f1c9c07ee70 2024-04-26T20:24:16.285 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.914+0000 7f1ca0c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:16.285 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.914+0000 7f1ca0c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37600/0 (socket says 172.21.15.102:37600) 2024-04-26T20:24:16.287 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.914+0000 7f1ca0c00700 1 -- 172.21.15.102:0/2910913882 learned_addr learned my addr 172.21.15.102:0/2910913882 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:16.287 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.915+0000 7f1ca0c00700 1 -- 172.21.15.102:0/2910913882 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c19f3b0 con 0x7f1c9c07ee70 2024-04-26T20:24:16.287 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.915+0000 7f1ca0c00700 1 --2- 172.21.15.102:0/2910913882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c19ee70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1c900116e0 tx=0x7f1c90011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:16.287 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.915+0000 7f1c99600700 1 -- 172.21.15.102:0/2910913882 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c90003a60 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.915+0000 7f1c99600700 1 -- 172.21.15.102:0/2910913882 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c9001b8f0 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.915+0000 7f1ca3600700 1 -- 172.21.15.102:0/2910913882 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c19f640 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.915+0000 7f1ca3600700 1 -- 172.21.15.102:0/2910913882 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c1a2330 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.915+0000 7f1c99600700 1 -- 172.21.15.102:0/2910913882 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c90019860 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.916+0000 7f1c99600700 1 -- 172.21.15.102:0/2910913882 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1c90019b10 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.917+0000 7f1ca3600700 1 -- 172.21.15.102:0/2910913882 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c9c064a80 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.917+0000 7f1c99600700 1 --2- 172.21.15.102:0/2910913882 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f1c840395b0 0x7f1c8403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.917+0000 7f1c99600700 1 -- 172.21.15.102:0/2910913882 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1c9004dd40 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.917+0000 7f1c9be00700 1 --2- 172.21.15.102:0/2910913882 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f1c840395b0 0x7f1c8403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.918+0000 7f1c9be00700 1 --2- 172.21.15.102:0/2910913882 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f1c840395b0 0x7f1c8403ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1c88006fd0 tx=0x7f1c88006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:14.923+0000 7f1c99600700 1 -- 172.21.15.102:0/2910913882 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1c90054080 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.153+0000 7f1ca3600700 1 -- 172.21.15.102:0/2910913882 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f1c9c1a28e0 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.156+0000 7f1c99600700 1 -- 172.21.15.102:0/2910913882 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f1c9001f030 con 0x7f1c9c07ee70 2024-04-26T20:24:16.288 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.159+0000 7f1ca3600700 1 -- 172.21.15.102:0/2910913882 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f1c840395b0 msgr2=0x7f1c8403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:16.290 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.159+0000 7f1ca3600700 1 --2- 172.21.15.102:0/2910913882 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f1c840395b0 0x7f1c8403ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1c88006fd0 tx=0x7f1c88006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:16.290 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.159+0000 7f1ca3600700 1 -- 172.21.15.102:0/2910913882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 msgr2=0x7f1c9c19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:16.290 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.159+0000 7f1ca3600700 1 --2- 172.21.15.102:0/2910913882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c19ee70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1c900116e0 tx=0x7f1c90011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:16.290 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.159+0000 7f1ca3600700 1 -- 172.21.15.102:0/2910913882 shutdown_connections 2024-04-26T20:24:16.290 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.159+0000 7f1ca3600700 1 --2- 172.21.15.102:0/2910913882 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f1c840395b0 0x7f1c8403ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:16.290 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.160+0000 7f1ca3600700 1 --2- 172.21.15.102:0/2910913882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c9c07ee70 0x7f1c9c19ee70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:16.290 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.160+0000 7f1ca3600700 1 -- 172.21.15.102:0/2910913882 >> 172.21.15.102:0/2910913882 conn(0x7f1c9c104370 msgr2=0x7f1c9c104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:16.290 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.160+0000 7f1ca3600700 1 -- 172.21.15.102:0/2910913882 shutdown_connections 2024-04-26T20:24:16.290 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:15.160+0000 7f1ca3600700 1 -- 172.21.15.102:0/2910913882 wait complete. 2024-04-26T20:24:16.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:16 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2910913882' entity='client.admin' 2024-04-26T20:24:16.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:16 smithi102 ceph-mon[28094]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:24:17.851 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:17 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:24:17.852 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:17 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2403108750' entity='client.admin' 2024-04-26T20:24:17.863 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.911+0000 7fa333c00700 1 Processor -- start 2024-04-26T20:24:17.863 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.911+0000 7fa333c00700 1 -- start start 2024-04-26T20:24:17.863 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.912+0000 7fa333c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c07d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:17.863 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.912+0000 7fa333c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c07d8c0 con 0x7fa32c07ee90 2024-04-26T20:24:17.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.912+0000 7fa331200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c07d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:17.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.912+0000 7fa331200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c07d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37614/0 (socket says 172.21.15.102:37614) 2024-04-26T20:24:17.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.912+0000 7fa331200700 1 -- 172.21.15.102:0/3482942018 learned_addr learned my addr 172.21.15.102:0/3482942018 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:17.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.913+0000 7fa331200700 1 -- 172.21.15.102:0/3482942018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32c07da40 con 0x7fa32c07ee90 2024-04-26T20:24:17.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.913+0000 7fa331200700 1 --2- 172.21.15.102:0/3482942018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c07d2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa3200098a0 tx=0x7fa320009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a75562b110542ff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:17.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.913+0000 7fa32b400700 1 -- 172.21.15.102:0/3482942018 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa320004160 con 0x7fa32c07ee90 2024-04-26T20:24:17.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.913+0000 7fa32b400700 1 -- 172.21.15.102:0/3482942018 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa320004300 con 0x7fa32c07ee90 2024-04-26T20:24:17.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.913+0000 7fa32b400700 1 -- 172.21.15.102:0/3482942018 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa320011a20 con 0x7fa32c07ee90 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.914+0000 7fa333c00700 1 -- 172.21.15.102:0/3482942018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 msgr2=0x7fa32c07d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.914+0000 7fa333c00700 1 --2- 172.21.15.102:0/3482942018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c07d2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa3200098a0 tx=0x7fa320009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.914+0000 7fa333c00700 1 -- 172.21.15.102:0/3482942018 shutdown_connections 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.914+0000 7fa333c00700 1 --2- 172.21.15.102:0/3482942018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c07d2f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.914+0000 7fa333c00700 1 -- 172.21.15.102:0/3482942018 >> 172.21.15.102:0/3482942018 conn(0x7fa32c104370 msgr2=0x7fa32c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.914+0000 7fa333c00700 1 -- 172.21.15.102:0/3482942018 shutdown_connections 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.914+0000 7fa333c00700 1 -- 172.21.15.102:0/3482942018 wait complete. 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.916+0000 7fa333c00700 1 Processor -- start 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.916+0000 7fa333c00700 1 -- start start 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.917+0000 7fa333c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.917+0000 7fa333c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c10f370 con 0x7fa32c07ee90 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.917+0000 7fa331200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.917+0000 7fa331200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37624/0 (socket says 172.21.15.102:37624) 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.917+0000 7fa331200700 1 -- 172.21.15.102:0/2403108750 learned_addr learned my addr 172.21.15.102:0/2403108750 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.917+0000 7fa331200700 1 -- 172.21.15.102:0/2403108750 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32c19f400 con 0x7fa32c07ee90 2024-04-26T20:24:17.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.918+0000 7fa331200700 1 --2- 172.21.15.102:0/2403108750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c19eec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa3200116e0 tx=0x7fa320011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.918+0000 7fa329600700 1 -- 172.21.15.102:0/2403108750 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa3200039e0 con 0x7fa32c07ee90 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.918+0000 7fa329600700 1 -- 172.21.15.102:0/2403108750 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa32001b8f0 con 0x7fa32c07ee90 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.918+0000 7fa333c00700 1 -- 172.21.15.102:0/2403108750 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa32c19f690 con 0x7fa32c07ee90 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.918+0000 7fa329600700 1 -- 172.21.15.102:0/2403108750 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa320023760 con 0x7fa32c07ee90 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.918+0000 7fa333c00700 1 -- 172.21.15.102:0/2403108750 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa32c1a2380 con 0x7fa32c07ee90 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.919+0000 7fa329600700 1 -- 172.21.15.102:0/2403108750 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa32001b440 con 0x7fa32c07ee90 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.919+0000 7fa333c00700 1 -- 172.21.15.102:0/2403108750 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa32c064a80 con 0x7fa32c07ee90 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.920+0000 7fa329600700 1 --2- 172.21.15.102:0/2403108750 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fa314039560 0x7fa31403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.920+0000 7fa329600700 1 -- 172.21.15.102:0/2403108750 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa32004dfa0 con 0x7fa32c07ee90 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.920+0000 7fa32be00700 1 --2- 172.21.15.102:0/2403108750 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fa314039560 0x7fa31403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.921+0000 7fa32be00700 1 --2- 172.21.15.102:0/2403108750 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fa314039560 0x7fa31403ba20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa318006fd0 tx=0x7fa318006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:16.925+0000 7fa329600700 1 -- 172.21.15.102:0/2403108750 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa32001f030 con 0x7fa32c07ee90 2024-04-26T20:24:17.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.161+0000 7fa333c00700 1 -- 172.21.15.102:0/2403108750 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fa32c1a29a0 con 0x7fa32c07ee90 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.164+0000 7fa329600700 1 -- 172.21.15.102:0/2403108750 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fa32001bd50 con 0x7fa32c07ee90 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.166+0000 7fa333c00700 1 -- 172.21.15.102:0/2403108750 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fa314039560 msgr2=0x7fa31403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.166+0000 7fa333c00700 1 --2- 172.21.15.102:0/2403108750 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fa314039560 0x7fa31403ba20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa318006fd0 tx=0x7fa318006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.166+0000 7fa333c00700 1 -- 172.21.15.102:0/2403108750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 msgr2=0x7fa32c19eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.166+0000 7fa333c00700 1 --2- 172.21.15.102:0/2403108750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c19eec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa3200116e0 tx=0x7fa320011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.167+0000 7fa333c00700 1 -- 172.21.15.102:0/2403108750 shutdown_connections 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.167+0000 7fa333c00700 1 --2- 172.21.15.102:0/2403108750 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fa314039560 0x7fa31403ba20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.167+0000 7fa333c00700 1 --2- 172.21.15.102:0/2403108750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa32c07ee90 0x7fa32c19eec0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.167+0000 7fa333c00700 1 -- 172.21.15.102:0/2403108750 >> 172.21.15.102:0/2403108750 conn(0x7fa32c104370 msgr2=0x7fa32c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.167+0000 7fa333c00700 1 -- 172.21.15.102:0/2403108750 shutdown_connections 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:17.167+0000 7fa333c00700 1 -- 172.21.15.102:0/2403108750 wait complete. 2024-04-26T20:24:17.871 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling the dashboard module... 2024-04-26T20:24:18.765 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:18 smithi102 ceph-mon[28094]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:24:18.765 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:18 smithi102 ceph-mon[28094]: from='mgr.14120 172.21.15.102:0/4048938912' entity='mgr.x' 2024-04-26T20:24:19.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:19 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3240261520' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T20:24:19.887 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:19 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: ignoring --setuser ceph since I am not root 2024-04-26T20:24:19.888 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:19 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: ignoring --setgroup ceph since I am not root 2024-04-26T20:24:19.888 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:19 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:19.749+0000 7fc3654a5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T20:24:20.148 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:19 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:19.989+0000 7fc3654a5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:24:20.149 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:20 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:20.125+0000 7fc3654a5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T20:24:20.638 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:20 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3240261520' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T20:24:20.638 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:20 smithi102 ceph-mon[28094]: mgrmap e9: x(active, since 25s) 2024-04-26T20:24:20.638 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:20 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:20.526+0000 7fc3654a5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T20:24:20.898 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:20 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:20.636+0000 7fc3654a5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T20:24:21.112 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.457+0000 7f5a2ca00700 1 Processor -- start 2024-04-26T20:24:21.112 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.458+0000 7f5a2ca00700 1 -- start start 2024-04-26T20:24:21.112 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.458+0000 7f5a2ca00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2807d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:21.112 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.458+0000 7f5a2ca00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2807d880 con 0x7f5a2807ee50 2024-04-26T20:24:21.112 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.458+0000 7f5a25e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2807d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.459+0000 7f5a25e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2807d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37636/0 (socket says 172.21.15.102:37636) 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.459+0000 7f5a25e00700 1 -- 172.21.15.102:0/716666126 learned_addr learned my addr 172.21.15.102:0/716666126 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.459+0000 7f5a25e00700 1 -- 172.21.15.102:0/716666126 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a2807da00 con 0x7f5a2807ee50 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.459+0000 7f5a25e00700 1 --2- 172.21.15.102:0/716666126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2807d2b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5a180098a0 tx=0x7f5a18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e649493e91ade5eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.460+0000 7f5a24a00700 1 -- 172.21.15.102:0/716666126 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a18004160 con 0x7f5a2807ee50 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.460+0000 7f5a24a00700 1 -- 172.21.15.102:0/716666126 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a18004300 con 0x7f5a2807ee50 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.460+0000 7f5a24a00700 1 -- 172.21.15.102:0/716666126 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a18011a20 con 0x7f5a2807ee50 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.460+0000 7f5a2ca00700 1 -- 172.21.15.102:0/716666126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 msgr2=0x7f5a2807d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.460+0000 7f5a2ca00700 1 --2- 172.21.15.102:0/716666126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2807d2b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5a180098a0 tx=0x7f5a18009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.460+0000 7f5a2ca00700 1 -- 172.21.15.102:0/716666126 shutdown_connections 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.460+0000 7f5a2ca00700 1 --2- 172.21.15.102:0/716666126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2807d2b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.460+0000 7f5a2ca00700 1 -- 172.21.15.102:0/716666126 >> 172.21.15.102:0/716666126 conn(0x7f5a28104370 msgr2=0x7f5a28106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:21.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.461+0000 7f5a2ca00700 1 -- 172.21.15.102:0/716666126 shutdown_connections 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.461+0000 7f5a2ca00700 1 -- 172.21.15.102:0/716666126 wait complete. 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.462+0000 7f5a2ca00700 1 Processor -- start 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.462+0000 7f5a2ca00700 1 -- start start 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.462+0000 7f5a2ca00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.462+0000 7f5a2ca00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2810f350 con 0x7f5a2807ee50 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.463+0000 7f5a25e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.463+0000 7f5a25e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2819ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37652/0 (socket says 172.21.15.102:37652) 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.463+0000 7f5a25e00700 1 -- 172.21.15.102:0/3240261520 learned_addr learned my addr 172.21.15.102:0/3240261520 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.463+0000 7f5a25e00700 1 -- 172.21.15.102:0/3240261520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a2819f3c0 con 0x7f5a2807ee50 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.463+0000 7f5a25e00700 1 --2- 172.21.15.102:0/3240261520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2819ee80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a18000c00 tx=0x7f5a18011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.464+0000 7f5a1ea00700 1 -- 172.21.15.102:0/3240261520 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a18003aa0 con 0x7f5a2807ee50 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.464+0000 7f5a1ea00700 1 -- 172.21.15.102:0/3240261520 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a1801b8f0 con 0x7f5a2807ee50 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.464+0000 7f5a1ea00700 1 -- 172.21.15.102:0/3240261520 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a18023870 con 0x7f5a2807ee50 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.464+0000 7f5a2ca00700 1 -- 172.21.15.102:0/3240261520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a2819f650 con 0x7f5a2807ee50 2024-04-26T20:24:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.464+0000 7f5a2ca00700 1 -- 172.21.15.102:0/3240261520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a281a2340 con 0x7f5a2807ee50 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.465+0000 7f5a1ea00700 1 -- 172.21.15.102:0/3240261520 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5a18023b70 con 0x7f5a2807ee50 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.465+0000 7f5a2ca00700 1 -- 172.21.15.102:0/3240261520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a28064a80 con 0x7f5a2807ee50 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.466+0000 7f5a1ea00700 1 --2- 172.21.15.102:0/3240261520 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f5a0c0395b0 0x7f5a0c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.466+0000 7f5a1ea00700 1 -- 172.21.15.102:0/3240261520 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5a1804df10 con 0x7f5a2807ee50 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.466+0000 7f5a25400700 1 --2- 172.21.15.102:0/3240261520 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f5a0c0395b0 0x7f5a0c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.467+0000 7f5a25400700 1 --2- 172.21.15.102:0/3240261520 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f5a0c0395b0 0x7f5a0c03ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5a14006fd0 tx=0x7f5a14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.472+0000 7f5a1ea00700 1 -- 172.21.15.102:0/3240261520 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5a1801f030 con 0x7f5a2807ee50 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:18.764+0000 7f5a2ca00700 1 -- 172.21.15.102:0/3240261520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f5a281a2510 con 0x7f5a2807ee50 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.432+0000 7f5a1ea00700 1 -- 172.21.15.102:0/3240261520 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f5a1801a3d0 con 0x7f5a2807ee50 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.434+0000 7f5a1ea00700 1 -- 172.21.15.102:0/3240261520 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f5a1800b3e0 con 0x7f5a2807ee50 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.438+0000 7f5a2ca00700 1 -- 172.21.15.102:0/3240261520 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f5a0c0395b0 msgr2=0x7f5a0c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.438+0000 7f5a2ca00700 1 --2- 172.21.15.102:0/3240261520 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f5a0c0395b0 0x7f5a0c03ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5a14006fd0 tx=0x7f5a14006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.438+0000 7f5a2ca00700 1 -- 172.21.15.102:0/3240261520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 msgr2=0x7f5a2819ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:21.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.438+0000 7f5a2ca00700 1 --2- 172.21.15.102:0/3240261520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2819ee80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a18000c00 tx=0x7f5a18011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:21.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.439+0000 7f5a2ca00700 1 -- 172.21.15.102:0/3240261520 shutdown_connections 2024-04-26T20:24:21.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.439+0000 7f5a2ca00700 1 --2- 172.21.15.102:0/3240261520 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f5a0c0395b0 0x7f5a0c03ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:21.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.439+0000 7f5a2ca00700 1 --2- 172.21.15.102:0/3240261520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2807ee50 0x7f5a2819ee80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:21.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.439+0000 7f5a2ca00700 1 -- 172.21.15.102:0/3240261520 >> 172.21.15.102:0/3240261520 conn(0x7f5a28104370 msgr2=0x7f5a28104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:21.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.439+0000 7f5a2ca00700 1 -- 172.21.15.102:0/3240261520 shutdown_connections 2024-04-26T20:24:21.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:19.439+0000 7f5a2ca00700 1 -- 172.21.15.102:0/3240261520 wait complete. 2024-04-26T20:24:22.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:22 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3675888238' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T20:24:22.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:22 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:22.239+0000 7fc3654a5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T20:24:22.780 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:22 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:22.473+0000 7fc3654a5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T20:24:22.780 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:22 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:22.701+0000 7fc3654a5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T20:24:22.827 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.775+0000 7f96ce200700 1 Processor -- start 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.776+0000 7f96ce200700 1 -- start start 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.776+0000 7f96ce200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c807d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.776+0000 7f96ce200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f96c807d8c0 con 0x7f96c807ee90 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.777+0000 7f96c7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c807d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.777+0000 7f96c7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c807d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37674/0 (socket says 172.21.15.102:37674) 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.777+0000 7f96c7400700 1 -- 172.21.15.102:0/1391260878 learned_addr learned my addr 172.21.15.102:0/1391260878 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.777+0000 7f96c7400700 1 -- 172.21.15.102:0/1391260878 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c807da40 con 0x7f96c807ee90 2024-04-26T20:24:22.828 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.778+0000 7f96c7400700 1 --2- 172.21.15.102:0/1391260878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c807d2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f96b80098a0 tx=0x7f96b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fbd586cb0fccfb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.778+0000 7f96c6000700 1 -- 172.21.15.102:0/1391260878 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96b8004160 con 0x7f96c807ee90 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.778+0000 7f96c6000700 1 -- 172.21.15.102:0/1391260878 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96b8004300 con 0x7f96c807ee90 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.778+0000 7f96c6000700 1 -- 172.21.15.102:0/1391260878 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96b8011a20 con 0x7f96c807ee90 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.779+0000 7f96ce200700 1 -- 172.21.15.102:0/1391260878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 msgr2=0x7f96c807d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.779+0000 7f96ce200700 1 --2- 172.21.15.102:0/1391260878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c807d2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f96b80098a0 tx=0x7f96b8009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.779+0000 7f96ce200700 1 -- 172.21.15.102:0/1391260878 shutdown_connections 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.779+0000 7f96ce200700 1 --2- 172.21.15.102:0/1391260878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c807d2f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.779+0000 7f96ce200700 1 -- 172.21.15.102:0/1391260878 >> 172.21.15.102:0/1391260878 conn(0x7f96c8104370 msgr2=0x7f96c8106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.779+0000 7f96ce200700 1 -- 172.21.15.102:0/1391260878 shutdown_connections 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.779+0000 7f96ce200700 1 -- 172.21.15.102:0/1391260878 wait complete. 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.780+0000 7f96ce200700 1 Processor -- start 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.781+0000 7f96ce200700 1 -- start start 2024-04-26T20:24:22.829 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.781+0000 7f96ce200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.781+0000 7f96ce200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f96c810f370 con 0x7f96c807ee90 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.781+0000 7f96c7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.781+0000 7f96c7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c819eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37680/0 (socket says 172.21.15.102:37680) 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.781+0000 7f96c7400700 1 -- 172.21.15.102:0/3675888238 learned_addr learned my addr 172.21.15.102:0/3675888238 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.782+0000 7f96c7400700 1 -- 172.21.15.102:0/3675888238 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c819f400 con 0x7f96c807ee90 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.782+0000 7f96c7400700 1 --2- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c819eec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009520 tx=0x7f96b8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.782+0000 7f96bfe00700 1 -- 172.21.15.102:0/3675888238 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96b80039e0 con 0x7f96c807ee90 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.782+0000 7f96bfe00700 1 -- 172.21.15.102:0/3675888238 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96b801b8f0 con 0x7f96c807ee90 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.782+0000 7f96ce200700 1 -- 172.21.15.102:0/3675888238 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c819f690 con 0x7f96c807ee90 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.782+0000 7f96ce200700 1 -- 172.21.15.102:0/3675888238 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c81a2380 con 0x7f96c807ee90 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.782+0000 7f96bfe00700 1 -- 172.21.15.102:0/3675888238 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96b8019770 con 0x7f96c807ee90 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.783+0000 7f96bfe00700 1 -- 172.21.15.102:0/3675888238 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f96b801b440 con 0x7f96c807ee90 2024-04-26T20:24:22.830 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.784+0000 7f96ce200700 1 -- 172.21.15.102:0/3675888238 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96c8064a80 con 0x7f96c807ee90 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.784+0000 7f96bfe00700 1 --2- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f96ac039600 0x7f96ac03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.784+0000 7f96bfe00700 1 -- 172.21.15.102:0/3675888238 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f96b804dfc0 con 0x7f96c807ee90 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.784+0000 7f96c6a00700 1 -- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f96ac039600 msgr2=0x7f96ac03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1441707916 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.784+0000 7f96c6a00700 1 --2- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f96ac039600 0x7f96ac03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.790+0000 7f96bfe00700 1 -- 172.21.15.102:0/3675888238 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f96b801f030 con 0x7f96c807ee90 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.985+0000 7f96c6a00700 1 -- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f96ac039600 msgr2=0x7f96ac03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1441707916 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:21.985+0000 7f96c6a00700 1 --2- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f96ac039600 0x7f96ac03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.089+0000 7f96ce200700 1 -- 172.21.15.102:0/3675888238 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f96c81a2970 con 0x7f96c807ee90 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.089+0000 7f96bfe00700 1 -- 172.21.15.102:0/3675888238 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f96b804d910 con 0x7f96c807ee90 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.092+0000 7f96ce200700 1 -- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f96ac039600 msgr2=0x7f96ac03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.092+0000 7f96ce200700 1 --2- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f96ac039600 0x7f96ac03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.092+0000 7f96ce200700 1 -- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 msgr2=0x7f96c819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.092+0000 7f96ce200700 1 --2- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c819eec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009520 tx=0x7f96b8011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.093+0000 7f96ce200700 1 -- 172.21.15.102:0/3675888238 shutdown_connections 2024-04-26T20:24:22.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.093+0000 7f96ce200700 1 --2- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7f96ac039600 0x7f96ac03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:22.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.093+0000 7f96ce200700 1 --2- 172.21.15.102:0/3675888238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96c807ee90 0x7f96c819eec0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:22.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.093+0000 7f96ce200700 1 -- 172.21.15.102:0/3675888238 >> 172.21.15.102:0/3675888238 conn(0x7f96c8104370 msgr2=0x7f96c8104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:22.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.093+0000 7f96ce200700 1 -- 172.21.15.102:0/3675888238 shutdown_connections 2024-04-26T20:24:22.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:22.093+0000 7f96ce200700 1 -- 172.21.15.102:0/3675888238 wait complete. 2024-04-26T20:24:22.832 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-04-26T20:24:22.832 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 9... 2024-04-26T20:24:23.102 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:23 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:23.100+0000 7fc3654a5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T20:24:23.382 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:23 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:23.229+0000 7fc3654a5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T20:24:23.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:23 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:23.502+0000 7fc3654a5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T20:24:24.057 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:23 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:23.764+0000 7fc3654a5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:24:24.857 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:24.539+0000 7fc3654a5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T20:24:25.148 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:25 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:25.078+0000 7fc3654a5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T20:24:25.463 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:25 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:25.203+0000 7fc3654a5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T20:24:25.463 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:25 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:25.462+0000 7fc3654a5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T20:24:26.398 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:26.098+0000 7fc3654a5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T20:24:26.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:26.219+0000 7fc3654a5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T20:24:26.834 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:26.533+0000 7fc3654a5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T20:24:26.834 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:26.707+0000 7fc3654a5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T20:24:26.834 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:26.832+0000 7fc3654a5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T20:24:27.148 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:26.982+0000 7fc3654a5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T20:24:27.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:27 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:27.344+0000 7fc3654a5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T20:24:28.374 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:28.086+0000 7fc3654a5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T20:24:28.648 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:24:28.372+0000 7fc3654a5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T20:24:28.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-mon[28094]: Active manager daemon x restarted 2024-04-26T20:24:28.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-mon[28094]: Activating manager daemon x 2024-04-26T20:24:28.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-mon[28094]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T20:24:28.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-mon[28094]: mgrmap e10: x(active, starting, since 0.00823276s) 2024-04-26T20:24:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:24:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T20:24:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T20:24:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T20:24:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T20:24:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:28 smithi102 ceph-mon[28094]: Manager daemon x is now available 2024-04-26T20:24:29.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:29.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T20:24:29.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T20:24:29.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:29 smithi102 ceph-mon[28094]: mgrmap e11: x(active, since 1.01041s) 2024-04-26T20:24:30.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:30 smithi102 ceph-mon[28094]: [26/Apr/2024:20:24:28] ENGINE Bus STARTING 2024-04-26T20:24:30.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:30 smithi102 ceph-mon[28094]: [26/Apr/2024:20:24:28] ENGINE Serving on http://172.21.15.102:8765 2024-04-26T20:24:30.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:30 smithi102 ceph-mon[28094]: [26/Apr/2024:20:24:29] ENGINE Serving on https://172.21.15.102:7150 2024-04-26T20:24:30.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:30 smithi102 ceph-mon[28094]: [26/Apr/2024:20:24:29] ENGINE Bus STARTED 2024-04-26T20:24:30.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:30 smithi102 ceph-mon[28094]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T20:24:30.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:30 smithi102 ceph-mon[28094]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T20:24:30.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-26T20:24:30.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T20:24:30.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T20:24:30.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-26T20:24:30.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.445+0000 7fb345600700 1 Processor -- start 2024-04-26T20:24:30.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.445+0000 7fb345600700 1 -- start start 2024-04-26T20:24:30.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.445+0000 7fb345600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb340109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:30.757 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.445+0000 7fb345600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb340109760 con 0x7fb340108d70 2024-04-26T20:24:30.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.446+0000 7fb33ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb340109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:30.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.446+0000 7fb33ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb340109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37682/0 (socket says 172.21.15.102:37682) 2024-04-26T20:24:30.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.446+0000 7fb33ea00700 1 -- 172.21.15.102:0/497436732 learned_addr learned my addr 172.21.15.102:0/497436732 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:30.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.446+0000 7fb33ea00700 1 -- 172.21.15.102:0/497436732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb340109f70 con 0x7fb340108d70 2024-04-26T20:24:30.764 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.447+0000 7fb33ea00700 1 --2- 172.21.15.102:0/497436732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb340109190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb3300098a0 tx=0x7fb330009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c636adaf499eeb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:30.764 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.447+0000 7fb33d600700 1 -- 172.21.15.102:0/497436732 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb330004160 con 0x7fb340108d70 2024-04-26T20:24:30.764 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.447+0000 7fb33d600700 1 -- 172.21.15.102:0/497436732 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb330004300 con 0x7fb340108d70 2024-04-26T20:24:30.764 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.447+0000 7fb33d600700 1 -- 172.21.15.102:0/497436732 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb330011a20 con 0x7fb340108d70 2024-04-26T20:24:30.765 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.448+0000 7fb345600700 1 -- 172.21.15.102:0/497436732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 msgr2=0x7fb340109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:30.765 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.448+0000 7fb345600700 1 --2- 172.21.15.102:0/497436732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb340109190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb3300098a0 tx=0x7fb330009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:30.765 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.448+0000 7fb345600700 1 -- 172.21.15.102:0/497436732 shutdown_connections 2024-04-26T20:24:30.765 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.448+0000 7fb345600700 1 --2- 172.21.15.102:0/497436732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb340109190 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:30.765 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.448+0000 7fb345600700 1 -- 172.21.15.102:0/497436732 >> 172.21.15.102:0/497436732 conn(0x7fb340104370 msgr2=0x7fb340106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:30.765 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.448+0000 7fb345600700 1 -- 172.21.15.102:0/497436732 shutdown_connections 2024-04-26T20:24:30.765 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.448+0000 7fb345600700 1 -- 172.21.15.102:0/497436732 wait complete. 2024-04-26T20:24:30.765 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.449+0000 7fb345600700 1 Processor -- start 2024-04-26T20:24:30.765 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.449+0000 7fb345600700 1 -- start start 2024-04-26T20:24:30.765 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.450+0000 7fb345600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb34019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:30.766 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.450+0000 7fb345600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb34010d320 con 0x7fb340108d70 2024-04-26T20:24:30.766 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.450+0000 7fb33ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb34019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:30.766 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.450+0000 7fb33ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb34019ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37688/0 (socket says 172.21.15.102:37688) 2024-04-26T20:24:30.766 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.450+0000 7fb33ea00700 1 -- 172.21.15.102:0/1285164117 learned_addr learned my addr 172.21.15.102:0/1285164117 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:30.766 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.450+0000 7fb33ea00700 1 -- 172.21.15.102:0/1285164117 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb34019f3a0 con 0x7fb340108d70 2024-04-26T20:24:30.766 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.451+0000 7fb33ea00700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb34019ee60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb330009520 tx=0x7fb330011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:30.766 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.451+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3300039e0 con 0x7fb340108d70 2024-04-26T20:24:30.766 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.451+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb33001b8f0 con 0x7fb340108d70 2024-04-26T20:24:30.766 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.451+0000 7fb345600700 1 -- 172.21.15.102:0/1285164117 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb34019f630 con 0x7fb340108d70 2024-04-26T20:24:30.766 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.451+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb330019770 con 0x7fb340108d70 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.451+0000 7fb345600700 1 -- 172.21.15.102:0/1285164117 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3401a2320 con 0x7fb340108d70 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.453+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fb33001b440 con 0x7fb340108d70 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.453+0000 7fb337400700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 0x7fb32403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.453+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 --> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb32403c220 con 0x7fb324039650 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.453+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb33004df70 con 0x7fb340108d70 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.453+0000 7fb33e000700 1 -- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 msgr2=0x7fb32403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1441707916 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.453+0000 7fb33e000700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 0x7fb32403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.654+0000 7fb33e000700 1 -- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 msgr2=0x7fb32403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1441707916 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:23.654+0000 7fb33e000700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 0x7fb32403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:24.055+0000 7fb33e000700 1 -- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 msgr2=0x7fb32403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1441707916 2024-04-26T20:24:30.767 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:24.055+0000 7fb33e000700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 0x7fb32403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:24.856+0000 7fb33e000700 1 -- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 msgr2=0x7fb32403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1441707916 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:24.856+0000 7fb33e000700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 0x7fb32403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:26.458+0000 7fb33e000700 1 -- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 msgr2=0x7fb32403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1441707916 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:26.458+0000 7fb33e000700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 0x7fb32403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:28.381+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7fb330017cf0 con 0x7fb340108d70 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:28.381+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 msgr2=0x7fb32403bb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:28.381+0000 7fb337400700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 0x7fb32403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.383+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7fb33004fe50 con 0x7fb340108d70 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.383+0000 7fb337400700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb32403cfe0 0x7fb32403f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.383+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb32403c220 con 0x7fb32403cfe0 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.384+0000 7fb33e000700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb32403cfe0 0x7fb32403f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.385+0000 7fb33e000700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb32403cfe0 0x7fb32403f3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb32c003a10 tx=0x7fb32c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.387+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fb32403c220 con 0x7fb32403cfe0 2024-04-26T20:24:30.768 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.396+0000 7fb345600700 1 -- 172.21.15.102:0/1285164117 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fb34006a540 con 0x7fb32403cfe0 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb337400700 1 -- 172.21.15.102:0/1285164117 <== mgr.14152 v2:172.21.15.102:6800/71145250 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fb34006a540 con 0x7fb32403cfe0 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 -- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb32403cfe0 msgr2=0x7fb32403f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb32403cfe0 0x7fb32403f3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb32c003a10 tx=0x7fb32c009270 comp rx=0 tx=0).stop 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 -- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 msgr2=0x7fb34019ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb34019ee60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb330009520 tx=0x7fb330011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 -- 172.21.15.102:0/1285164117 shutdown_connections 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb32403cfe0 0x7fb32403f3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb340108d70 0x7fb34019ee60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 --2- 172.21.15.102:0/1285164117 >> [v2:172.21.15.102:6800/1441707916,v1:172.21.15.102:6801/1441707916] conn(0x7fb324039650 0x7fb32403bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 -- 172.21.15.102:0/1285164117 >> 172.21.15.102:0/1285164117 conn(0x7fb340104370 msgr2=0x7fb340104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 -- 172.21.15.102:0/1285164117 shutdown_connections 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:29.397+0000 7fb345600700 1 -- 172.21.15.102:0/1285164117 wait complete. 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 9 is available 2024-04-26T20:24:30.769 INFO:teuthology.orchestra.run.smithi102.stdout:Generating a dashboard self-signed certificate... 2024-04-26T20:24:31.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:31 smithi102 ceph-mon[28094]: mgrmap e12: x(active, since 2s) 2024-04-26T20:24:31.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:32.573 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T20:24:32.573 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.432+0000 7f09a1600700 1 Processor -- start 2024-04-26T20:24:32.573 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.432+0000 7f09a1600700 1 -- start start 2024-04-26T20:24:32.573 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.432+0000 7f09a1600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:32.573 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.432+0000 7f09a1600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f099c109760 con 0x7f099c108d70 2024-04-26T20:24:32.573 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.433+0000 7f099aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:32.573 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.433+0000 7f099aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49808/0 (socket says 172.21.15.102:49808) 2024-04-26T20:24:32.573 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.433+0000 7f099aa00700 1 -- 172.21.15.102:0/1261403757 learned_addr learned my addr 172.21.15.102:0/1261403757 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:32.574 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.433+0000 7f099aa00700 1 -- 172.21.15.102:0/1261403757 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099c109f70 con 0x7f099c108d70 2024-04-26T20:24:32.579 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.434+0000 7f099aa00700 1 --2- 172.21.15.102:0/1261403757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c109190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f098c0098a0 tx=0x7f098c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a7c704155a4cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:32.579 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.434+0000 7f0999600700 1 -- 172.21.15.102:0/1261403757 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f098c004160 con 0x7f099c108d70 2024-04-26T20:24:32.579 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.434+0000 7f0999600700 1 -- 172.21.15.102:0/1261403757 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f098c004300 con 0x7f099c108d70 2024-04-26T20:24:32.580 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.434+0000 7f0999600700 1 -- 172.21.15.102:0/1261403757 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f098c011a20 con 0x7f099c108d70 2024-04-26T20:24:32.580 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.435+0000 7f09a1600700 1 -- 172.21.15.102:0/1261403757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 msgr2=0x7f099c109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:32.580 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.435+0000 7f09a1600700 1 --2- 172.21.15.102:0/1261403757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c109190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f098c0098a0 tx=0x7f098c009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:32.580 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.435+0000 7f09a1600700 1 -- 172.21.15.102:0/1261403757 shutdown_connections 2024-04-26T20:24:32.580 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.435+0000 7f09a1600700 1 --2- 172.21.15.102:0/1261403757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c109190 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:32.580 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.435+0000 7f09a1600700 1 -- 172.21.15.102:0/1261403757 >> 172.21.15.102:0/1261403757 conn(0x7f099c104370 msgr2=0x7f099c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.435+0000 7f09a1600700 1 -- 172.21.15.102:0/1261403757 shutdown_connections 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.435+0000 7f09a1600700 1 -- 172.21.15.102:0/1261403757 wait complete. 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.436+0000 7f09a1600700 1 Processor -- start 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.436+0000 7f09a1600700 1 -- start start 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.437+0000 7f09a1600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.437+0000 7f09a1600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f099c10d320 con 0x7f099c108d70 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.437+0000 7f099aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.437+0000 7f099aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49820/0 (socket says 172.21.15.102:49820) 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.437+0000 7f099aa00700 1 -- 172.21.15.102:0/1535020796 learned_addr learned my addr 172.21.15.102:0/1535020796 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.437+0000 7f099aa00700 1 -- 172.21.15.102:0/1535020796 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099c19f400 con 0x7f099c108d70 2024-04-26T20:24:32.581 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.438+0000 7f099aa00700 1 --2- 172.21.15.102:0/1535020796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c19eec0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f098c009520 tx=0x7f098c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:32.582 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.438+0000 7f0993400700 1 -- 172.21.15.102:0/1535020796 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f098c0039e0 con 0x7f099c108d70 2024-04-26T20:24:32.582 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.438+0000 7f0993400700 1 -- 172.21.15.102:0/1535020796 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f098c01b8f0 con 0x7f099c108d70 2024-04-26T20:24:32.582 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.438+0000 7f0993400700 1 -- 172.21.15.102:0/1535020796 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f098c019770 con 0x7f099c108d70 2024-04-26T20:24:32.582 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.438+0000 7f09a1600700 1 -- 172.21.15.102:0/1535020796 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f099c19f690 con 0x7f099c108d70 2024-04-26T20:24:32.582 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.438+0000 7f09a1600700 1 -- 172.21.15.102:0/1535020796 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f099c1a2380 con 0x7f099c108d70 2024-04-26T20:24:32.582 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.439+0000 7f0993400700 1 -- 172.21.15.102:0/1535020796 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f098c019a20 con 0x7f099c108d70 2024-04-26T20:24:32.582 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.439+0000 7f09a1600700 1 -- 172.21.15.102:0/1535020796 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f099c064a80 con 0x7f099c108d70 2024-04-26T20:24:32.582 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.440+0000 7f0993400700 1 --2- 172.21.15.102:0/1535020796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0980039650 0x7f098003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:32.582 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.440+0000 7f0993400700 1 -- 172.21.15.102:0/1535020796 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f098c04e1b0 con 0x7f099c108d70 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.440+0000 7f099a000700 1 --2- 172.21.15.102:0/1535020796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0980039650 0x7f098003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.441+0000 7f099a000700 1 --2- 172.21.15.102:0/1535020796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0980039650 0x7f098003bb10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0988006fd0 tx=0x7f0988006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.446+0000 7f0993400700 1 -- 172.21.15.102:0/1535020796 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f098c01f030 con 0x7f099c108d70 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.678+0000 7f09a1600700 1 -- 172.21.15.102:0/1535020796 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f099c1054c0 con 0x7f0980039650 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.841+0000 7f0993400700 1 -- 172.21.15.102:0/1535020796 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f099c1054c0 con 0x7f0980039650 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.843+0000 7f09a1600700 1 -- 172.21.15.102:0/1535020796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0980039650 msgr2=0x7f098003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.844+0000 7f09a1600700 1 --2- 172.21.15.102:0/1535020796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0980039650 0x7f098003bb10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0988006fd0 tx=0x7f0988006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.844+0000 7f09a1600700 1 -- 172.21.15.102:0/1535020796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 msgr2=0x7f099c19eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.844+0000 7f09a1600700 1 --2- 172.21.15.102:0/1535020796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c19eec0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f098c009520 tx=0x7f098c011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.844+0000 7f09a1600700 1 -- 172.21.15.102:0/1535020796 shutdown_connections 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.844+0000 7f09a1600700 1 --2- 172.21.15.102:0/1535020796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0980039650 0x7f098003bb10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.844+0000 7f09a1600700 1 --2- 172.21.15.102:0/1535020796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f099c108d70 0x7f099c19eec0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:32.583 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.844+0000 7f09a1600700 1 -- 172.21.15.102:0/1535020796 >> 172.21.15.102:0/1535020796 conn(0x7f099c104370 msgr2=0x7f099c104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:32.584 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.844+0000 7f09a1600700 1 -- 172.21.15.102:0/1535020796 shutdown_connections 2024-04-26T20:24:32.584 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:31.844+0000 7f09a1600700 1 -- 172.21.15.102:0/1535020796 wait complete. 2024-04-26T20:24:32.584 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial admin user... 2024-04-26T20:24:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:32 smithi102 ceph-mon[28094]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:24:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:34.851 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$pAiic96UAOqeZ0RmvrKcTuZzqBZ3pqfbw2K6062vuTVZoyvj/SkKW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714163073, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T20:24:34.851 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.321+0000 7f951c000700 1 Processor -- start 2024-04-26T20:24:34.853 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.321+0000 7f951c000700 1 -- start start 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.322+0000 7f951c000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f951410bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.322+0000 7f951c000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f951407ce80 con 0x7f9514109830 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.322+0000 7f9519600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f951410bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.322+0000 7f9519600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f951410bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49830/0 (socket says 172.21.15.102:49830) 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.322+0000 7f9519600700 1 -- 172.21.15.102:0/3282542505 learned_addr learned my addr 172.21.15.102:0/3282542505 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.323+0000 7f9519600700 1 -- 172.21.15.102:0/3282542505 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951410c160 con 0x7f9514109830 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.323+0000 7f9519600700 1 --2- 172.21.15.102:0/3282542505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f951410bc20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f95080098a0 tx=0x7f9508009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c22117ef0d8e40 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.323+0000 7f9513e00700 1 -- 172.21.15.102:0/3282542505 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9508004160 con 0x7f9514109830 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.323+0000 7f9513e00700 1 -- 172.21.15.102:0/3282542505 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9508004300 con 0x7f9514109830 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.323+0000 7f9513e00700 1 -- 172.21.15.102:0/3282542505 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9508011a20 con 0x7f9514109830 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.324+0000 7f951c000700 1 -- 172.21.15.102:0/3282542505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 msgr2=0x7f951410bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:34.854 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.324+0000 7f951c000700 1 --2- 172.21.15.102:0/3282542505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f951410bc20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f95080098a0 tx=0x7f9508009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:34.858 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.324+0000 7f951c000700 1 -- 172.21.15.102:0/3282542505 shutdown_connections 2024-04-26T20:24:34.858 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.324+0000 7f951c000700 1 --2- 172.21.15.102:0/3282542505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f951410bc20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:34.858 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.324+0000 7f951c000700 1 -- 172.21.15.102:0/3282542505 >> 172.21.15.102:0/3282542505 conn(0x7f9514103240 msgr2=0x7f9514105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:34.858 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.324+0000 7f951c000700 1 -- 172.21.15.102:0/3282542505 shutdown_connections 2024-04-26T20:24:34.858 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.325+0000 7f951c000700 1 -- 172.21.15.102:0/3282542505 wait complete. 2024-04-26T20:24:34.858 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.326+0000 7f951c000700 1 Processor -- start 2024-04-26T20:24:34.858 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.326+0000 7f951c000700 1 -- start start 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.326+0000 7f951c000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f95141090b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.326+0000 7f951c000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f951410d160 con 0x7f9514109830 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.326+0000 7f9519600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f95141090b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.326+0000 7f9519600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f95141090b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49840/0 (socket says 172.21.15.102:49840) 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.326+0000 7f9519600700 1 -- 172.21.15.102:0/3812506399 learned_addr learned my addr 172.21.15.102:0/3812506399 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.327+0000 7f9519600700 1 -- 172.21.15.102:0/3812506399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95141095f0 con 0x7f9514109830 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.327+0000 7f9519600700 1 --2- 172.21.15.102:0/3812506399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f95141090b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9508004660 tx=0x7f9508011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.328+0000 7f9512000700 1 -- 172.21.15.102:0/3812506399 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9508003aa0 con 0x7f9514109830 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.328+0000 7f9512000700 1 -- 172.21.15.102:0/3812506399 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f950801b8f0 con 0x7f9514109830 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.328+0000 7f951c000700 1 -- 172.21.15.102:0/3812506399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95141077e0 con 0x7f9514109830 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.328+0000 7f9512000700 1 -- 172.21.15.102:0/3812506399 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9508023900 con 0x7f9514109830 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.328+0000 7f951c000700 1 -- 172.21.15.102:0/3812506399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9514107d20 con 0x7f9514109830 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.329+0000 7f951c000700 1 -- 172.21.15.102:0/3812506399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9514064a80 con 0x7f9514109830 2024-04-26T20:24:34.859 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.329+0000 7f9512000700 1 -- 172.21.15.102:0/3812506399 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f950801baa0 con 0x7f9514109830 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.330+0000 7f9512000700 1 --2- 172.21.15.102:0/3812506399 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94fc039660 0x7f94fc03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.330+0000 7f9512000700 1 -- 172.21.15.102:0/3812506399 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f950804dfc0 con 0x7f9514109830 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.330+0000 7f9518c00700 1 --2- 172.21.15.102:0/3812506399 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94fc039660 0x7f94fc03bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.331+0000 7f9518c00700 1 --2- 172.21.15.102:0/3812506399 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94fc039660 0x7f94fc03bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9500006fd0 tx=0x7f9500006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.336+0000 7f9512000700 1 -- 172.21.15.102:0/3812506399 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f950801f030 con 0x7f9514109830 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.653+0000 7f951c000700 1 -- 172.21.15.102:0/3812506399 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f9514108580 con 0x7f94fc039660 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.894+0000 7f9512000700 1 -- 172.21.15.102:0/3812506399 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f9514108580 con 0x7f94fc039660 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.897+0000 7f951c000700 1 -- 172.21.15.102:0/3812506399 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94fc039660 msgr2=0x7f94fc03bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.897+0000 7f951c000700 1 --2- 172.21.15.102:0/3812506399 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94fc039660 0x7f94fc03bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9500006fd0 tx=0x7f9500006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.897+0000 7f951c000700 1 -- 172.21.15.102:0/3812506399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 msgr2=0x7f95141090b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:34.860 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.897+0000 7f951c000700 1 --2- 172.21.15.102:0/3812506399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f95141090b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9508004660 tx=0x7f9508011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:34.861 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.897+0000 7f951c000700 1 -- 172.21.15.102:0/3812506399 shutdown_connections 2024-04-26T20:24:34.861 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.897+0000 7f951c000700 1 --2- 172.21.15.102:0/3812506399 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94fc039660 0x7f94fc03bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:34.861 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.897+0000 7f951c000700 1 --2- 172.21.15.102:0/3812506399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9514109830 0x7f95141090b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:34.861 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.897+0000 7f951c000700 1 -- 172.21.15.102:0/3812506399 >> 172.21.15.102:0/3812506399 conn(0x7f9514103240 msgr2=0x7f9514103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:34.862 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.898+0000 7f951c000700 1 -- 172.21.15.102:0/3812506399 shutdown_connections 2024-04-26T20:24:34.862 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:33.898+0000 7f951c000700 1 -- 172.21.15.102:0/3812506399 wait complete. 2024-04-26T20:24:34.862 INFO:teuthology.orchestra.run.smithi102.stdout:Fetching dashboard port number... 2024-04-26T20:24:35.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:34 smithi102 ceph-mon[28094]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:24:35.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:36.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:35 smithi102 ceph-mon[28094]: mgrmap e13: x(active, since 6s) 2024-04-26T20:24:36.913 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T20:24:36.913 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.976+0000 7f36ffe00700 1 Processor -- start 2024-04-26T20:24:36.913 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.976+0000 7f36ffe00700 1 -- start start 2024-04-26T20:24:36.913 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.976+0000 7f36ffe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f810b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:36.913 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.977+0000 7f36ffe00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f36f810b960 con 0x7f36f810af70 2024-04-26T20:24:36.913 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.977+0000 7f36fd400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f810b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:36.913 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.977+0000 7f36fd400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f810b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51066/0 (socket says 172.21.15.102:51066) 2024-04-26T20:24:36.913 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.977+0000 7f36fd400700 1 -- 172.21.15.102:0/2377125780 learned_addr learned my addr 172.21.15.102:0/2377125780 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:36.913 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.977+0000 7f36fd400700 1 -- 172.21.15.102:0/2377125780 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f810c170 con 0x7f36f810af70 2024-04-26T20:24:36.913 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.978+0000 7f36fd400700 1 --2- 172.21.15.102:0/2377125780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f810b390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f36ec0098a0 tx=0x7f36ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e63a22fc4cddfe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.978+0000 7f36f7e00700 1 -- 172.21.15.102:0/2377125780 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36ec004160 con 0x7f36f810af70 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.978+0000 7f36f7e00700 1 -- 172.21.15.102:0/2377125780 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36ec004300 con 0x7f36f810af70 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.978+0000 7f36f7e00700 1 -- 172.21.15.102:0/2377125780 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36ec011a20 con 0x7f36f810af70 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.979+0000 7f36ffe00700 1 -- 172.21.15.102:0/2377125780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 msgr2=0x7f36f810b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.979+0000 7f36ffe00700 1 --2- 172.21.15.102:0/2377125780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f810b390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f36ec0098a0 tx=0x7f36ec009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.979+0000 7f36ffe00700 1 -- 172.21.15.102:0/2377125780 shutdown_connections 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.979+0000 7f36ffe00700 1 --2- 172.21.15.102:0/2377125780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f810b390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.979+0000 7f36ffe00700 1 -- 172.21.15.102:0/2377125780 >> 172.21.15.102:0/2377125780 conn(0x7f36f807e370 msgr2=0x7f36f807e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.979+0000 7f36ffe00700 1 -- 172.21.15.102:0/2377125780 shutdown_connections 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.979+0000 7f36ffe00700 1 -- 172.21.15.102:0/2377125780 wait complete. 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.980+0000 7f36ffe00700 1 Processor -- start 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.980+0000 7f36ffe00700 1 -- start start 2024-04-26T20:24:36.914 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.980+0000 7f36ffe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.980+0000 7f36ffe00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f36f810f520 con 0x7f36f810af70 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.981+0000 7f36fd400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.981+0000 7f36fd400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f819eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51080/0 (socket says 172.21.15.102:51080) 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.981+0000 7f36fd400700 1 -- 172.21.15.102:0/1349057198 learned_addr learned my addr 172.21.15.102:0/1349057198 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.981+0000 7f36fd400700 1 -- 172.21.15.102:0/1349057198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f819f400 con 0x7f36f810af70 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.981+0000 7f36fd400700 1 --2- 172.21.15.102:0/1349057198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f819eec0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f36ec000c00 tx=0x7f36ec011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.982+0000 7f36f6000700 1 -- 172.21.15.102:0/1349057198 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36ec003aa0 con 0x7f36f810af70 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.982+0000 7f36f6000700 1 -- 172.21.15.102:0/1349057198 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36ec01b8f0 con 0x7f36f810af70 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.982+0000 7f36f6000700 1 -- 172.21.15.102:0/1349057198 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36ec023890 con 0x7f36f810af70 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.982+0000 7f36ffe00700 1 -- 172.21.15.102:0/1349057198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f819f690 con 0x7f36f810af70 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.982+0000 7f36ffe00700 1 -- 172.21.15.102:0/1349057198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f81a2380 con 0x7f36f810af70 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.983+0000 7f36f6000700 1 -- 172.21.15.102:0/1349057198 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f36ec01b440 con 0x7f36f810af70 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.983+0000 7f36ffe00700 1 -- 172.21.15.102:0/1349057198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f8064a80 con 0x7f36f810af70 2024-04-26T20:24:36.915 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.983+0000 7f36f6000700 1 --2- 172.21.15.102:0/1349057198 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f36e0039700 0x7f36e003bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.983+0000 7f36f6000700 1 -- 172.21.15.102:0/1349057198 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f36ec04e100 con 0x7f36f810af70 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.984+0000 7f36fca00700 1 --2- 172.21.15.102:0/1349057198 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f36e0039700 0x7f36e003bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.984+0000 7f36fca00700 1 --2- 172.21.15.102:0/1349057198 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f36e0039700 0x7f36e003bbc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f36e4006fd0 tx=0x7f36e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:35.989+0000 7f36f6000700 1 -- 172.21.15.102:0/1349057198 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f36ec01f030 con 0x7f36f810af70 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.215+0000 7f36ffe00700 1 -- 172.21.15.102:0/1349057198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f36f81a2550 con 0x7f36f810af70 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.216+0000 7f36f6000700 1 -- 172.21.15.102:0/1349057198 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f36ec0189e0 con 0x7f36f810af70 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.219+0000 7f36ffe00700 1 -- 172.21.15.102:0/1349057198 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f36e0039700 msgr2=0x7f36e003bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.219+0000 7f36ffe00700 1 --2- 172.21.15.102:0/1349057198 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f36e0039700 0x7f36e003bbc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f36e4006fd0 tx=0x7f36e4006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.220+0000 7f36ffe00700 1 -- 172.21.15.102:0/1349057198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 msgr2=0x7f36f819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.220+0000 7f36ffe00700 1 --2- 172.21.15.102:0/1349057198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f819eec0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f36ec000c00 tx=0x7f36ec011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.220+0000 7f36ffe00700 1 -- 172.21.15.102:0/1349057198 shutdown_connections 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.220+0000 7f36ffe00700 1 --2- 172.21.15.102:0/1349057198 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f36e0039700 0x7f36e003bbc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.220+0000 7f36ffe00700 1 --2- 172.21.15.102:0/1349057198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36f810af70 0x7f36f819eec0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:36.916 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.220+0000 7f36ffe00700 1 -- 172.21.15.102:0/1349057198 >> 172.21.15.102:0/1349057198 conn(0x7f36f807e370 msgr2=0x7f36f81084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:36.917 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.220+0000 7f36ffe00700 1 -- 172.21.15.102:0/1349057198 shutdown_connections 2024-04-26T20:24:36.917 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:36.220+0000 7f36ffe00700 1 -- 172.21.15.102:0/1349057198 wait complete. 2024-04-26T20:24:36.944 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T20:24:36.944 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-04-26T20:24:36.967 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T20:24:36.967 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-26T20:24:36.967 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-04-26T20:24:36.967 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T20:24:36.971 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph Dashboard is now available at: 2024-04-26T20:24:36.971 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:24:36.971 INFO:teuthology.orchestra.run.smithi102.stdout: URL: https://smithi102.front.sepia.ceph.com:8443/ 2024-04-26T20:24:36.971 INFO:teuthology.orchestra.run.smithi102.stdout: User: admin 2024-04-26T20:24:36.971 INFO:teuthology.orchestra.run.smithi102.stdout: Password: v8kwi79zwl 2024-04-26T20:24:36.971 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:24:36.972 INFO:teuthology.orchestra.run.smithi102.stdout:Saving cluster configuration to /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config directory 2024-04-26T20:24:36.973 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling autotune for osd_memory_target 2024-04-26T20:24:37.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:36 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1349057198' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T20:24:38.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.559+0000 7f11faa00700 1 Processor -- start 2024-04-26T20:24:38.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.559+0000 7f11faa00700 1 -- start start 2024-04-26T20:24:38.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.560+0000 7f11faa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f407d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:38.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.560+0000 7f11faa00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f11f407d8c0 con 0x7f11f407ee90 2024-04-26T20:24:38.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.560+0000 7f11f3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f407d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:38.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.560+0000 7f11f3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f407d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51084/0 (socket says 172.21.15.102:51084) 2024-04-26T20:24:38.536 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.560+0000 7f11f3e00700 1 -- 172.21.15.102:0/291434802 learned_addr learned my addr 172.21.15.102:0/291434802 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.561+0000 7f11f3e00700 1 -- 172.21.15.102:0/291434802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f407da40 con 0x7f11f407ee90 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.561+0000 7f11f3e00700 1 --2- 172.21.15.102:0/291434802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f407d2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f11e40098a0 tx=0x7f11e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b624038c143c727d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.561+0000 7f11f2a00700 1 -- 172.21.15.102:0/291434802 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11e4004160 con 0x7f11f407ee90 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.562+0000 7f11f2a00700 1 -- 172.21.15.102:0/291434802 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11e4004300 con 0x7f11f407ee90 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.562+0000 7f11f2a00700 1 -- 172.21.15.102:0/291434802 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11e4011a20 con 0x7f11f407ee90 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.562+0000 7f11faa00700 1 -- 172.21.15.102:0/291434802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 msgr2=0x7f11f407d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.562+0000 7f11faa00700 1 --2- 172.21.15.102:0/291434802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f407d2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f11e40098a0 tx=0x7f11e4009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.562+0000 7f11faa00700 1 -- 172.21.15.102:0/291434802 shutdown_connections 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.562+0000 7f11faa00700 1 --2- 172.21.15.102:0/291434802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f407d2f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.562+0000 7f11faa00700 1 -- 172.21.15.102:0/291434802 >> 172.21.15.102:0/291434802 conn(0x7f11f4104370 msgr2=0x7f11f4106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.563+0000 7f11faa00700 1 -- 172.21.15.102:0/291434802 shutdown_connections 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.563+0000 7f11faa00700 1 -- 172.21.15.102:0/291434802 wait complete. 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.564+0000 7f11faa00700 1 Processor -- start 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.564+0000 7f11faa00700 1 -- start start 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.564+0000 7f11faa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:38.537 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.564+0000 7f11faa00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f11f410f370 con 0x7f11f407ee90 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.564+0000 7f11f3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.564+0000 7f11f3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f419eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51098/0 (socket says 172.21.15.102:51098) 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.564+0000 7f11f3e00700 1 -- 172.21.15.102:0/2522903431 learned_addr learned my addr 172.21.15.102:0/2522903431 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.565+0000 7f11f3e00700 1 -- 172.21.15.102:0/2522903431 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f419f400 con 0x7f11f407ee90 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.565+0000 7f11f3e00700 1 --2- 172.21.15.102:0/2522903431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f419eec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f11e40116e0 tx=0x7f11e4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.565+0000 7f11f0c00700 1 -- 172.21.15.102:0/2522903431 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11e40039e0 con 0x7f11f407ee90 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.565+0000 7f11f0c00700 1 -- 172.21.15.102:0/2522903431 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11e401b8f0 con 0x7f11f407ee90 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.565+0000 7f11f0c00700 1 -- 172.21.15.102:0/2522903431 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11e4023760 con 0x7f11f407ee90 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.565+0000 7f11faa00700 1 -- 172.21.15.102:0/2522903431 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f419f690 con 0x7f11f407ee90 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.566+0000 7f11faa00700 1 -- 172.21.15.102:0/2522903431 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f41a2380 con 0x7f11f407ee90 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.567+0000 7f11f0c00700 1 -- 172.21.15.102:0/2522903431 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f11e401b440 con 0x7f11f407ee90 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.567+0000 7f11faa00700 1 -- 172.21.15.102:0/2522903431 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f4064a80 con 0x7f11f407ee90 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.567+0000 7f11f0c00700 1 --2- 172.21.15.102:0/2522903431 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f11d80396b0 0x7f11d803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:38.538 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.567+0000 7f11f0c00700 1 -- 172.21.15.102:0/2522903431 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f11e404e070 con 0x7f11f407ee90 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.568+0000 7f11f3400700 1 --2- 172.21.15.102:0/2522903431 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f11d80396b0 0x7f11d803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.568+0000 7f11f3400700 1 --2- 172.21.15.102:0/2522903431 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f11d80396b0 0x7f11d803bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f11e0006fd0 tx=0x7f11e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.573+0000 7f11f0c00700 1 -- 172.21.15.102:0/2522903431 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f11e401f030 con 0x7f11f407ee90 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.799+0000 7f11faa00700 1 -- 172.21.15.102:0/2522903431 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f11f41a29a0 con 0x7f11f407ee90 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.800+0000 7f11f0c00700 1 -- 172.21.15.102:0/2522903431 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f11e402bb90 con 0x7f11f407ee90 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.802+0000 7f11faa00700 1 -- 172.21.15.102:0/2522903431 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f11d80396b0 msgr2=0x7f11d803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.802+0000 7f11faa00700 1 --2- 172.21.15.102:0/2522903431 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f11d80396b0 0x7f11d803bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f11e0006fd0 tx=0x7f11e0006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.802+0000 7f11faa00700 1 -- 172.21.15.102:0/2522903431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 msgr2=0x7f11f419eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.802+0000 7f11faa00700 1 --2- 172.21.15.102:0/2522903431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f419eec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f11e40116e0 tx=0x7f11e4011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.803+0000 7f11faa00700 1 -- 172.21.15.102:0/2522903431 shutdown_connections 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.803+0000 7f11faa00700 1 --2- 172.21.15.102:0/2522903431 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f11d80396b0 0x7f11d803bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.803+0000 7f11faa00700 1 --2- 172.21.15.102:0/2522903431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11f407ee90 0x7f11f419eec0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.803+0000 7f11faa00700 1 -- 172.21.15.102:0/2522903431 >> 172.21.15.102:0/2522903431 conn(0x7f11f4104370 msgr2=0x7f11f4104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:38.539 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.803+0000 7f11faa00700 1 -- 172.21.15.102:0/2522903431 shutdown_connections 2024-04-26T20:24:38.540 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:37.803+0000 7f11faa00700 1 -- 172.21.15.102:0/2522903431 wait complete. 2024-04-26T20:24:40.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.115+0000 7faef0000700 1 Processor -- start 2024-04-26T20:24:40.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.115+0000 7faef0000700 1 -- start start 2024-04-26T20:24:40.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.116+0000 7faef0000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee8109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:40.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.116+0000 7faef0000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faee8109760 con 0x7faee8108d70 2024-04-26T20:24:40.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.116+0000 7faeed600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee8109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:40.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.116+0000 7faeed600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee8109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51114/0 (socket says 172.21.15.102:51114) 2024-04-26T20:24:40.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.116+0000 7faeed600700 1 -- 172.21.15.102:0/898138998 learned_addr learned my addr 172.21.15.102:0/898138998 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:40.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.117+0000 7faeed600700 1 -- 172.21.15.102:0/898138998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee8109f70 con 0x7faee8108d70 2024-04-26T20:24:40.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.117+0000 7faeed600700 1 --2- 172.21.15.102:0/898138998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee8109190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faedc0098a0 tx=0x7faedc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89cded78360d6595 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:40.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.117+0000 7faee7e00700 1 -- 172.21.15.102:0/898138998 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faedc004160 con 0x7faee8108d70 2024-04-26T20:24:40.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.118+0000 7faee7e00700 1 -- 172.21.15.102:0/898138998 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faedc004300 con 0x7faee8108d70 2024-04-26T20:24:40.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.118+0000 7faee7e00700 1 -- 172.21.15.102:0/898138998 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faedc011a20 con 0x7faee8108d70 2024-04-26T20:24:40.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.118+0000 7faef0000700 1 -- 172.21.15.102:0/898138998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 msgr2=0x7faee8109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:40.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.118+0000 7faef0000700 1 --2- 172.21.15.102:0/898138998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee8109190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faedc0098a0 tx=0x7faedc009bb0 comp rx=0 tx=0).stop 2024-04-26T20:24:40.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.119+0000 7faef0000700 1 -- 172.21.15.102:0/898138998 shutdown_connections 2024-04-26T20:24:40.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.119+0000 7faef0000700 1 --2- 172.21.15.102:0/898138998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee8109190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:40.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.119+0000 7faef0000700 1 -- 172.21.15.102:0/898138998 >> 172.21.15.102:0/898138998 conn(0x7faee8104370 msgr2=0x7faee8106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.119+0000 7faef0000700 1 -- 172.21.15.102:0/898138998 shutdown_connections 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.119+0000 7faef0000700 1 -- 172.21.15.102:0/898138998 wait complete. 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.120+0000 7faef0000700 1 Processor -- start 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.120+0000 7faef0000700 1 -- start start 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.120+0000 7faef0000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.120+0000 7faef0000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faee810d320 con 0x7faee8108d70 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.120+0000 7faeed600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.121+0000 7faeed600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee819eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51116/0 (socket says 172.21.15.102:51116) 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.121+0000 7faeed600700 1 -- 172.21.15.102:0/963154287 learned_addr learned my addr 172.21.15.102:0/963154287 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.121+0000 7faeed600700 1 -- 172.21.15.102:0/963154287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee819f400 con 0x7faee8108d70 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.121+0000 7faeed600700 1 --2- 172.21.15.102:0/963154287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee819eec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faedc009520 tx=0x7faedc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:40.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.121+0000 7faee6000700 1 -- 172.21.15.102:0/963154287 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faedc0039e0 con 0x7faee8108d70 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.122+0000 7faee6000700 1 -- 172.21.15.102:0/963154287 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faedc01b8f0 con 0x7faee8108d70 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.122+0000 7faef0000700 1 -- 172.21.15.102:0/963154287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee819f690 con 0x7faee8108d70 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.122+0000 7faee6000700 1 -- 172.21.15.102:0/963154287 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faedc019770 con 0x7faee8108d70 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.122+0000 7faef0000700 1 -- 172.21.15.102:0/963154287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee81a2380 con 0x7faee8108d70 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.123+0000 7faee6000700 1 -- 172.21.15.102:0/963154287 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7faedc01b440 con 0x7faee8108d70 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.123+0000 7faef0000700 1 -- 172.21.15.102:0/963154287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faee8064a80 con 0x7faee8108d70 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.123+0000 7faee6000700 1 --2- 172.21.15.102:0/963154287 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faed00396f0 0x7faed003bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.123+0000 7faee6000700 1 -- 172.21.15.102:0/963154287 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faedc04e160 con 0x7faee8108d70 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.124+0000 7faeecc00700 1 --2- 172.21.15.102:0/963154287 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faed00396f0 0x7faed003bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.124+0000 7faeecc00700 1 --2- 172.21.15.102:0/963154287 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faed00396f0 0x7faed003bbb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faed8006fd0 tx=0x7faed8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:40.125 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.129+0000 7faee6000700 1 -- 172.21.15.102:0/963154287 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faedc01f030 con 0x7faee8108d70 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.470+0000 7faef0000700 1 -- 172.21.15.102:0/963154287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7faee81a2970 con 0x7faee8108d70 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.475+0000 7faee6000700 1 -- 172.21.15.102:0/963154287 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7faedc050160 con 0x7faee8108d70 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.478+0000 7faef0000700 1 -- 172.21.15.102:0/963154287 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faed00396f0 msgr2=0x7faed003bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.478+0000 7faef0000700 1 --2- 172.21.15.102:0/963154287 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faed00396f0 0x7faed003bbb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faed8006fd0 tx=0x7faed8006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.478+0000 7faef0000700 1 -- 172.21.15.102:0/963154287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 msgr2=0x7faee819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.478+0000 7faef0000700 1 --2- 172.21.15.102:0/963154287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee819eec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faedc009520 tx=0x7faedc011aa0 comp rx=0 tx=0).stop 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.478+0000 7faef0000700 1 -- 172.21.15.102:0/963154287 shutdown_connections 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.478+0000 7faef0000700 1 --2- 172.21.15.102:0/963154287 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faed00396f0 0x7faed003bbb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.478+0000 7faef0000700 1 --2- 172.21.15.102:0/963154287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faee8108d70 0x7faee819eec0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.478+0000 7faef0000700 1 -- 172.21.15.102:0/963154287 >> 172.21.15.102:0/963154287 conn(0x7faee8104370 msgr2=0x7faee8104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.479+0000 7faef0000700 1 -- 172.21.15.102:0/963154287 shutdown_connections 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-26T20:24:39.479+0000 7faef0000700 1 -- 172.21.15.102:0/963154287 wait complete. 2024-04-26T20:24:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T20:24:40.127 INFO:teuthology.orchestra.run.smithi102.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T20:24:40.127 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:24:40.127 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:24:40.127 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:24:40.127 INFO:teuthology.orchestra.run.smithi102.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T20:24:40.127 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:24:40.127 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T20:24:40.128 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:24:40.128 INFO:teuthology.orchestra.run.smithi102.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T20:24:40.128 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:24:40.128 INFO:teuthology.orchestra.run.smithi102.stdout: ceph telemetry on 2024-04-26T20:24:40.128 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:24:40.128 INFO:teuthology.orchestra.run.smithi102.stdout:For more information see: 2024-04-26T20:24:40.128 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:24:40.128 INFO:teuthology.orchestra.run.smithi102.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T20:24:40.128 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:24:40.128 INFO:teuthology.orchestra.run.smithi102.stdout:Bootstrap complete. 2024-04-26T20:24:40.231 INFO:tasks.cephadm:Fetching config... 2024-04-26T20:24:40.231 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:24:40.231 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T20:24:40.280 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T20:24:40.280 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:24:40.280 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T20:24:40.362 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T20:24:40.363 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:24:40.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T20:24:40.478 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T20:24:40.479 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:24:40.479 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T20:24:40.573 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T20:24:40.573 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfx7g9Ew1nCB0wLtv6SVsQ621pqMc/t/JIRUPBeX7P/aZK0xLM2ec8mVL/qn4XekcfMMaPNqhprCI8C9zgevKYoYRWVds+UftjnKYOqpy89WXVw2n9LSWlJYvVr8A7FhvcHcelSlnTaeMkuli31C/DH+y0ZqdFxm81uJo/0nCw6ZZulhjMbkZDjWzq5IFiDUQ7uSWo428HzWMCKgt+fM4IhGCWZVbRXSaexo8CeMMqadrrQISsHBnIEQv2ni9sNyKHQhD8L6iVBAtq7o1SnlomupaY9G098jRYrE3YLQ6xtqKG6lTR7NHUEQAOuc5IsqlRfnP70cFlb/gnoYkkJzmgkiCUvz2uK82d/YiZKLhT+NII7++0QbNsPf6A64Dm8kTzvICec3Ws9DQ1bn+2KOniieLbPAsL3ZS2wWIoRRbaxCHE6VNeqkrntxRkIsIZ01NrLtutAB/4W9bbCvU1RPwcO4WGnELGlKCiszqfeKJGodxJD6fu8/ZHwzPws0SPKr0= ceph-a94c830a-040a-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T20:24:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfx7g9Ew1nCB0wLtv6SVsQ621pqMc/t/JIRUPBeX7P/aZK0xLM2ec8mVL/qn4XekcfMMaPNqhprCI8C9zgevKYoYRWVds+UftjnKYOqpy89WXVw2n9LSWlJYvVr8A7FhvcHcelSlnTaeMkuli31C/DH+y0ZqdFxm81uJo/0nCw6ZZulhjMbkZDjWzq5IFiDUQ7uSWo428HzWMCKgt+fM4IhGCWZVbRXSaexo8CeMMqadrrQISsHBnIEQv2ni9sNyKHQhD8L6iVBAtq7o1SnlomupaY9G098jRYrE3YLQ6xtqKG6lTR7NHUEQAOuc5IsqlRfnP70cFlb/gnoYkkJzmgkiCUvz2uK82d/YiZKLhT+NII7++0QbNsPf6A64Dm8kTzvICec3Ws9DQ1bn+2KOniieLbPAsL3ZS2wWIoRRbaxCHE6VNeqkrntxRkIsIZ01NrLtutAB/4W9bbCvU1RPwcO4WGnELGlKCiszqfeKJGodxJD6fu8/ZHwzPws0SPKr0= ceph-a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:24:40.701 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:40 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/963154287' entity='client.admin' 2024-04-26T20:24:40.703 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfx7g9Ew1nCB0wLtv6SVsQ621pqMc/t/JIRUPBeX7P/aZK0xLM2ec8mVL/qn4XekcfMMaPNqhprCI8C9zgevKYoYRWVds+UftjnKYOqpy89WXVw2n9LSWlJYvVr8A7FhvcHcelSlnTaeMkuli31C/DH+y0ZqdFxm81uJo/0nCw6ZZulhjMbkZDjWzq5IFiDUQ7uSWo428HzWMCKgt+fM4IhGCWZVbRXSaexo8CeMMqadrrQISsHBnIEQv2ni9sNyKHQhD8L6iVBAtq7o1SnlomupaY9G098jRYrE3YLQ6xtqKG6lTR7NHUEQAOuc5IsqlRfnP70cFlb/gnoYkkJzmgkiCUvz2uK82d/YiZKLhT+NII7++0QbNsPf6A64Dm8kTzvICec3Ws9DQ1bn+2KOniieLbPAsL3ZS2wWIoRRbaxCHE6VNeqkrntxRkIsIZ01NrLtutAB/4W9bbCvU1RPwcO4WGnELGlKCiszqfeKJGodxJD6fu8/ZHwzPws0SPKr0= ceph-a94c830a-040a-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T20:24:40.741 INFO:teuthology.orchestra.run.smithi112.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfx7g9Ew1nCB0wLtv6SVsQ621pqMc/t/JIRUPBeX7P/aZK0xLM2ec8mVL/qn4XekcfMMaPNqhprCI8C9zgevKYoYRWVds+UftjnKYOqpy89WXVw2n9LSWlJYvVr8A7FhvcHcelSlnTaeMkuli31C/DH+y0ZqdFxm81uJo/0nCw6ZZulhjMbkZDjWzq5IFiDUQ7uSWo428HzWMCKgt+fM4IhGCWZVbRXSaexo8CeMMqadrrQISsHBnIEQv2ni9sNyKHQhD8L6iVBAtq7o1SnlomupaY9G098jRYrE3YLQ6xtqKG6lTR7NHUEQAOuc5IsqlRfnP70cFlb/gnoYkkJzmgkiCUvz2uK82d/YiZKLhT+NII7++0QbNsPf6A64Dm8kTzvICec3Ws9DQ1bn+2KOniieLbPAsL3ZS2wWIoRRbaxCHE6VNeqkrntxRkIsIZ01NrLtutAB/4W9bbCvU1RPwcO4WGnELGlKCiszqfeKJGodxJD6fu8/ZHwzPws0SPKr0= ceph-a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:24:40.753 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfx7g9Ew1nCB0wLtv6SVsQ621pqMc/t/JIRUPBeX7P/aZK0xLM2ec8mVL/qn4XekcfMMaPNqhprCI8C9zgevKYoYRWVds+UftjnKYOqpy89WXVw2n9LSWlJYvVr8A7FhvcHcelSlnTaeMkuli31C/DH+y0ZqdFxm81uJo/0nCw6ZZulhjMbkZDjWzq5IFiDUQ7uSWo428HzWMCKgt+fM4IhGCWZVbRXSaexo8CeMMqadrrQISsHBnIEQv2ni9sNyKHQhD8L6iVBAtq7o1SnlomupaY9G098jRYrE3YLQ6xtqKG6lTR7NHUEQAOuc5IsqlRfnP70cFlb/gnoYkkJzmgkiCUvz2uK82d/YiZKLhT+NII7++0QbNsPf6A64Dm8kTzvICec3Ws9DQ1bn+2KOniieLbPAsL3ZS2wWIoRRbaxCHE6VNeqkrntxRkIsIZ01NrLtutAB/4W9bbCvU1RPwcO4WGnELGlKCiszqfeKJGodxJD6fu8/ZHwzPws0SPKr0= ceph-a94c830a-040a-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T20:24:40.791 INFO:teuthology.orchestra.run.smithi130.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfx7g9Ew1nCB0wLtv6SVsQ621pqMc/t/JIRUPBeX7P/aZK0xLM2ec8mVL/qn4XekcfMMaPNqhprCI8C9zgevKYoYRWVds+UftjnKYOqpy89WXVw2n9LSWlJYvVr8A7FhvcHcelSlnTaeMkuli31C/DH+y0ZqdFxm81uJo/0nCw6ZZulhjMbkZDjWzq5IFiDUQ7uSWo428HzWMCKgt+fM4IhGCWZVbRXSaexo8CeMMqadrrQISsHBnIEQv2ni9sNyKHQhD8L6iVBAtq7o1SnlomupaY9G098jRYrE3YLQ6xtqKG6lTR7NHUEQAOuc5IsqlRfnP70cFlb/gnoYkkJzmgkiCUvz2uK82d/YiZKLhT+NII7++0QbNsPf6A64Dm8kTzvICec3Ws9DQ1bn+2KOniieLbPAsL3ZS2wWIoRRbaxCHE6VNeqkrntxRkIsIZ01NrLtutAB/4W9bbCvU1RPwcO4WGnELGlKCiszqfeKJGodxJD6fu8/ZHwzPws0SPKr0= ceph-a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:24:40.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T20:24:41.226 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:24:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.101+0000 7f9a1a200700 1 -- 172.21.15.102:0/3363848699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a14104e90 msgr2=0x7f9a141052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:44.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.101+0000 7f9a1a200700 1 --2- 172.21.15.102:0/3363848699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a14104e90 0x7f9a141052b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a08009910 tx=0x7f9a08009c20 comp rx=0 tx=0).stop 2024-04-26T20:24:44.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.101+0000 7f9a1a200700 1 -- 172.21.15.102:0/3363848699 shutdown_connections 2024-04-26T20:24:44.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.101+0000 7f9a1a200700 1 --2- 172.21.15.102:0/3363848699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a14104e90 0x7f9a141052b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:44.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.101+0000 7f9a1a200700 1 -- 172.21.15.102:0/3363848699 >> 172.21.15.102:0/3363848699 conn(0x7f9a14100480 msgr2=0x7f9a141028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:44.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.101+0000 7f9a1a200700 1 -- 172.21.15.102:0/3363848699 shutdown_connections 2024-04-26T20:24:44.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.101+0000 7f9a1a200700 1 -- 172.21.15.102:0/3363848699 wait complete. 2024-04-26T20:24:44.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.102+0000 7f9a1a200700 1 Processor -- start 2024-04-26T20:24:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.102+0000 7f9a1a200700 1 -- start start 2024-04-26T20:24:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.103+0000 7f9a1a200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a14104e90 0x7f9a1419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.103+0000 7f9a1a200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1410ad90 con 0x7f9a14104e90 2024-04-26T20:24:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.103+0000 7f9a18e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a14104e90 0x7f9a1419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.103+0000 7f9a18e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a14104e90 0x7f9a1419a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51124/0 (socket says 172.21.15.102:51124) 2024-04-26T20:24:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.103+0000 7f9a18e00700 1 -- 172.21.15.102:0/4200423215 learned_addr learned my addr 172.21.15.102:0/4200423215 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.104+0000 7f9a18e00700 1 -- 172.21.15.102:0/4200423215 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1419aa70 con 0x7f9a14104e90 2024-04-26T20:24:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.104+0000 7f9a18e00700 1 --2- 172.21.15.102:0/4200423215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a14104e90 0x7f9a1419a530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9a08005950 tx=0x7f9a08004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.104+0000 7f9a11600700 1 -- 172.21.15.102:0/4200423215 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a08026070 con 0x7f9a14104e90 2024-04-26T20:24:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.104+0000 7f9a11600700 1 -- 172.21.15.102:0/4200423215 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0800bc00 con 0x7f9a14104e90 2024-04-26T20:24:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.104+0000 7f9a1a200700 1 -- 172.21.15.102:0/4200423215 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a1419ad00 con 0x7f9a14104e90 2024-04-26T20:24:44.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.104+0000 7f9a11600700 1 -- 172.21.15.102:0/4200423215 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a08003a10 con 0x7f9a14104e90 2024-04-26T20:24:44.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.104+0000 7f9a1a200700 1 -- 172.21.15.102:0/4200423215 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a1419d9f0 con 0x7f9a14104e90 2024-04-26T20:24:44.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.105+0000 7f9a11600700 1 -- 172.21.15.102:0/4200423215 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f9a08029070 con 0x7f9a14104e90 2024-04-26T20:24:44.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.105+0000 7f9a1a200700 1 -- 172.21.15.102:0/4200423215 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a1410b510 con 0x7f9a14104e90 2024-04-26T20:24:44.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.106+0000 7f9a11600700 1 --2- 172.21.15.102:0/4200423215 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f99fc0396b0 0x7f99fc03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:44.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.106+0000 7f9a11600700 1 -- 172.21.15.102:0/4200423215 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9a0801e070 con 0x7f9a14104e90 2024-04-26T20:24:44.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.106+0000 7f9a13e00700 1 --2- 172.21.15.102:0/4200423215 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f99fc0396b0 0x7f99fc03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:44.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.107+0000 7f9a13e00700 1 --2- 172.21.15.102:0/4200423215 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f99fc0396b0 0x7f99fc03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a00006fd0 tx=0x7f9a00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:44.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.112+0000 7f9a11600700 1 -- 172.21.15.102:0/4200423215 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9a08013070 con 0x7f9a14104e90 2024-04-26T20:24:44.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.339+0000 7f9a1a200700 1 -- 172.21.15.102:0/4200423215 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f9a1402e5a0 con 0x7f9a14104e90 2024-04-26T20:24:44.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.342+0000 7f9a11600700 1 -- 172.21.15.102:0/4200423215 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f9a080300c0 con 0x7f9a14104e90 2024-04-26T20:24:44.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.345+0000 7f9a1a200700 1 -- 172.21.15.102:0/4200423215 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f99fc0396b0 msgr2=0x7f99fc03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:44.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.345+0000 7f9a1a200700 1 --2- 172.21.15.102:0/4200423215 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f99fc0396b0 0x7f99fc03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a00006fd0 tx=0x7f9a00006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:44.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.345+0000 7f9a1a200700 1 -- 172.21.15.102:0/4200423215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a14104e90 msgr2=0x7f9a1419a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:44.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.345+0000 7f9a1a200700 1 --2- 172.21.15.102:0/4200423215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a14104e90 0x7f9a1419a530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9a08005950 tx=0x7f9a08004fa0 comp rx=0 tx=0).stop 2024-04-26T20:24:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.346+0000 7f9a1a200700 1 -- 172.21.15.102:0/4200423215 shutdown_connections 2024-04-26T20:24:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.346+0000 7f9a1a200700 1 --2- 172.21.15.102:0/4200423215 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f99fc0396b0 0x7f99fc03bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.346+0000 7f9a1a200700 1 --2- 172.21.15.102:0/4200423215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a14104e90 0x7f9a1419a530 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.346+0000 7f9a1a200700 1 -- 172.21.15.102:0/4200423215 >> 172.21.15.102:0/4200423215 conn(0x7f9a14100480 msgr2=0x7f9a14100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.346+0000 7f9a1a200700 1 -- 172.21.15.102:0/4200423215 shutdown_connections 2024-04-26T20:24:44.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:44.346+0000 7f9a1a200700 1 -- 172.21.15.102:0/4200423215 wait complete. 2024-04-26T20:24:44.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:44.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:44.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:24:44.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:44.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:44.367 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:45.171 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T20:24:45.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T20:24:45.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:45 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/4200423215' entity='client.admin' 2024-04-26T20:24:45.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:45.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:45.577 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:24:48.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.119+0000 7f7784c00700 1 -- 172.21.15.102:0/2579094461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7780109630 msgr2=0x7f7780109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:48.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.119+0000 7f7784c00700 1 --2- 172.21.15.102:0/2579094461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7780109630 0x7f7780109a50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7770009910 tx=0x7f7770009c20 comp rx=0 tx=0).stop 2024-04-26T20:24:48.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.119+0000 7f7784c00700 1 -- 172.21.15.102:0/2579094461 shutdown_connections 2024-04-26T20:24:48.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.119+0000 7f7784c00700 1 --2- 172.21.15.102:0/2579094461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7780109630 0x7f7780109a50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:48.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.119+0000 7f7784c00700 1 -- 172.21.15.102:0/2579094461 >> 172.21.15.102:0/2579094461 conn(0x7f77800783e0 msgr2=0x7f778007a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:48.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.119+0000 7f7784c00700 1 -- 172.21.15.102:0/2579094461 shutdown_connections 2024-04-26T20:24:48.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.119+0000 7f7784c00700 1 -- 172.21.15.102:0/2579094461 wait complete. 2024-04-26T20:24:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.121+0000 7f7784c00700 1 Processor -- start 2024-04-26T20:24:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.121+0000 7f7784c00700 1 -- start start 2024-04-26T20:24:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.121+0000 7f7784c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7780109630 0x7f778019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.121+0000 7f7784c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f778010f200 con 0x7f7780109630 2024-04-26T20:24:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.121+0000 7f777f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7780109630 0x7f778019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.121+0000 7f777f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7780109630 0x7f778019e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35734/0 (socket says 172.21.15.102:35734) 2024-04-26T20:24:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.121+0000 7f777f400700 1 -- 172.21.15.102:0/1281738610 learned_addr learned my addr 172.21.15.102:0/1281738610 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.122+0000 7f777f400700 1 -- 172.21.15.102:0/1281738610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f778019ee50 con 0x7f7780109630 2024-04-26T20:24:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.122+0000 7f777f400700 1 --2- 172.21.15.102:0/1281738610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7780109630 0x7f778019e910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7770005800 tx=0x7f7770009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:48.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.122+0000 7f7777e00700 1 -- 172.21.15.102:0/1281738610 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7770027070 con 0x7f7780109630 2024-04-26T20:24:48.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.123+0000 7f7777e00700 1 -- 172.21.15.102:0/1281738610 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f777000be10 con 0x7f7780109630 2024-04-26T20:24:48.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.123+0000 7f7784c00700 1 -- 172.21.15.102:0/1281738610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f778019f0e0 con 0x7f7780109630 2024-04-26T20:24:48.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.123+0000 7f7777e00700 1 -- 172.21.15.102:0/1281738610 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f777000f460 con 0x7f7780109630 2024-04-26T20:24:48.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.123+0000 7f7784c00700 1 -- 172.21.15.102:0/1281738610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77801a1dd0 con 0x7f7780109630 2024-04-26T20:24:48.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.124+0000 7f7777e00700 1 -- 172.21.15.102:0/1281738610 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f777002a070 con 0x7f7780109630 2024-04-26T20:24:48.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.124+0000 7f7784c00700 1 -- 172.21.15.102:0/1281738610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f778010f980 con 0x7f7780109630 2024-04-26T20:24:48.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.124+0000 7f7777e00700 1 --2- 172.21.15.102:0/1281738610 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77640396b0 0x7f776403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:48.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.124+0000 7f7777e00700 1 -- 172.21.15.102:0/1281738610 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f777001e030 con 0x7f7780109630 2024-04-26T20:24:48.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.125+0000 7f777ea00700 1 --2- 172.21.15.102:0/1281738610 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77640396b0 0x7f776403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:48.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.125+0000 7f777ea00700 1 --2- 172.21.15.102:0/1281738610 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77640396b0 0x7f776403bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f776c006fd0 tx=0x7f776c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:48.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.130+0000 7f7777e00700 1 -- 172.21.15.102:0/1281738610 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7770013070 con 0x7f7780109630 2024-04-26T20:24:48.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.359+0000 7f7784c00700 1 -- 172.21.15.102:0/1281738610 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f778002e5a0 con 0x7f77640396b0 2024-04-26T20:24:48.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.364+0000 7f7777e00700 1 -- 172.21.15.102:0/1281738610 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f778002e5a0 con 0x7f77640396b0 2024-04-26T20:24:48.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.367+0000 7f7784c00700 1 -- 172.21.15.102:0/1281738610 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77640396b0 msgr2=0x7f776403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:48.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.367+0000 7f7784c00700 1 --2- 172.21.15.102:0/1281738610 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77640396b0 0x7f776403bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f776c006fd0 tx=0x7f776c006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:48.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.367+0000 7f7784c00700 1 -- 172.21.15.102:0/1281738610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7780109630 msgr2=0x7f778019e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:48.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.367+0000 7f7784c00700 1 --2- 172.21.15.102:0/1281738610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7780109630 0x7f778019e910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7770005800 tx=0x7f7770009f60 comp rx=0 tx=0).stop 2024-04-26T20:24:48.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.367+0000 7f7784c00700 1 -- 172.21.15.102:0/1281738610 shutdown_connections 2024-04-26T20:24:48.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.367+0000 7f7784c00700 1 --2- 172.21.15.102:0/1281738610 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77640396b0 0x7f776403bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:48.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.367+0000 7f7784c00700 1 --2- 172.21.15.102:0/1281738610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7780109630 0x7f778019e910 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:48.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.367+0000 7f7784c00700 1 -- 172.21.15.102:0/1281738610 >> 172.21.15.102:0/1281738610 conn(0x7f77800783e0 msgr2=0x7f7780078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:48.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.367+0000 7f7784c00700 1 -- 172.21.15.102:0/1281738610 shutdown_connections 2024-04-26T20:24:48.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:48.368+0000 7f7784c00700 1 -- 172.21.15.102:0/1281738610 wait complete. 2024-04-26T20:24:49.146 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi112 2024-04-26T20:24:49.146 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:24:49.146 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.conf 2024-04-26T20:24:49.165 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:24:49.165 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:24:49.222 INFO:tasks.cephadm:Adding host smithi112 to orchestrator... 2024-04-26T20:24:49.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch host add smithi112 2024-04-26T20:24:49.627 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:24:49.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:49 smithi102 ceph-mon[28094]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:24:49.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:49 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:49.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:49 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:49.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:49 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:24:49.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:49 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:24:49.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:49 smithi102 ceph-mon[28094]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-26T20:24:49.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:49 smithi102 ceph-mon[28094]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:24:50.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:50 smithi102 ceph-mon[28094]: Updating smithi102:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:24:51.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:51 smithi102 ceph-mon[28094]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:24:51.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:51 smithi102 ceph-mon[28094]: Updating smithi102:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T20:24:51.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:51 smithi102 ceph-mon[28094]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:24:51.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:51.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:51.557 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:51.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.723+0000 7f3a6ce00700 1 -- 172.21.15.102:0/1859168229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a68102680 msgr2=0x7f3a68102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:51.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.723+0000 7f3a6ce00700 1 --2- 172.21.15.102:0/1859168229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a68102680 0x7f3a68102a60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009910 tx=0x7f3a58009c20 comp rx=0 tx=0).stop 2024-04-26T20:24:51.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.723+0000 7f3a6ce00700 1 -- 172.21.15.102:0/1859168229 shutdown_connections 2024-04-26T20:24:51.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.723+0000 7f3a6ce00700 1 --2- 172.21.15.102:0/1859168229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a68102680 0x7f3a68102a60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:51.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.723+0000 7f3a6ce00700 1 -- 172.21.15.102:0/1859168229 >> 172.21.15.102:0/1859168229 conn(0x7f3a680776c0 msgr2=0x7f3a68077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:51.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.723+0000 7f3a6ce00700 1 -- 172.21.15.102:0/1859168229 shutdown_connections 2024-04-26T20:24:51.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.723+0000 7f3a6ce00700 1 -- 172.21.15.102:0/1859168229 wait complete. 2024-04-26T20:24:51.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.724+0000 7f3a6ce00700 1 Processor -- start 2024-04-26T20:24:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.724+0000 7f3a6ce00700 1 -- start start 2024-04-26T20:24:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.725+0000 7f3a6ce00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a681a07c0 0x7f3a681a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.725+0000 7f3a6ce00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6810f3a0 con 0x7f3a681a07c0 2024-04-26T20:24:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.725+0000 7f3a67400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a681a07c0 0x7f3a681a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.725+0000 7f3a67400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a681a07c0 0x7f3a681a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35752/0 (socket says 172.21.15.102:35752) 2024-04-26T20:24:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.725+0000 7f3a67400700 1 -- 172.21.15.102:0/235879202 learned_addr learned my addr 172.21.15.102:0/235879202 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:51.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.725+0000 7f3a67400700 1 -- 172.21.15.102:0/235879202 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a681a10e0 con 0x7f3a681a07c0 2024-04-26T20:24:51.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.726+0000 7f3a67400700 1 --2- 172.21.15.102:0/235879202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a681a07c0 0x7f3a681a0ba0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3a58005950 tx=0x7f3a58005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:51.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.726+0000 7f3a5fe00700 1 -- 172.21.15.102:0/235879202 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a58026070 con 0x7f3a681a07c0 2024-04-26T20:24:51.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.726+0000 7f3a5fe00700 1 -- 172.21.15.102:0/235879202 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a5800b750 con 0x7f3a681a07c0 2024-04-26T20:24:51.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.726+0000 7f3a6ce00700 1 -- 172.21.15.102:0/235879202 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a681a1370 con 0x7f3a681a07c0 2024-04-26T20:24:51.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.726+0000 7f3a5fe00700 1 -- 172.21.15.102:0/235879202 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a580037f0 con 0x7f3a681a07c0 2024-04-26T20:24:51.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.726+0000 7f3a6ce00700 1 -- 172.21.15.102:0/235879202 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a6819ab50 con 0x7f3a681a07c0 2024-04-26T20:24:51.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.727+0000 7f3a5fe00700 1 -- 172.21.15.102:0/235879202 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f3a58029070 con 0x7f3a681a07c0 2024-04-26T20:24:51.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.727+0000 7f3a6ce00700 1 -- 172.21.15.102:0/235879202 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a68102190 con 0x7f3a681a07c0 2024-04-26T20:24:51.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.728+0000 7f3a5fe00700 1 --2- 172.21.15.102:0/235879202 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3a4c0396b0 0x7f3a4c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:51.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.728+0000 7f3a5fe00700 1 -- 172.21.15.102:0/235879202 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3a5801e070 con 0x7f3a681a07c0 2024-04-26T20:24:51.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.728+0000 7f3a66a00700 1 --2- 172.21.15.102:0/235879202 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3a4c0396b0 0x7f3a4c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:51.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.729+0000 7f3a66a00700 1 --2- 172.21.15.102:0/235879202 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3a4c0396b0 0x7f3a4c03bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a54006fd0 tx=0x7f3a54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:51.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.733+0000 7f3a5fe00700 1 -- 172.21.15.102:0/235879202 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3a58013070 con 0x7f3a681a07c0 2024-04-26T20:24:51.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:51.967+0000 7f3a6ce00700 1 -- 172.21.15.102:0/235879202 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}) -- 0x7f3a6819add0 con 0x7f3a4c0396b0 2024-04-26T20:24:53.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:53 smithi102 ceph-mon[28094]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:24:53.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:53 smithi102 ceph-mon[28094]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:24:54.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:54 smithi102 ceph-mon[28094]: Deploying cephadm binary to smithi112 2024-04-26T20:24:55.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:55 smithi102 ceph-mon[28094]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:24:56.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.633+0000 7f3a5fe00700 1 -- 172.21.15.102:0/235879202 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f3a6819add0 con 0x7f3a4c0396b0 2024-04-26T20:24:56.635 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi112' with addr '172.21.15.112' 2024-04-26T20:24:56.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.637+0000 7f3a6ce00700 1 -- 172.21.15.102:0/235879202 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3a4c0396b0 msgr2=0x7f3a4c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:56.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.637+0000 7f3a6ce00700 1 --2- 172.21.15.102:0/235879202 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3a4c0396b0 0x7f3a4c03bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a54006fd0 tx=0x7f3a54006e40 comp rx=0 tx=0).stop 2024-04-26T20:24:56.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.637+0000 7f3a6ce00700 1 -- 172.21.15.102:0/235879202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a681a07c0 msgr2=0x7f3a681a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:56.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.637+0000 7f3a6ce00700 1 --2- 172.21.15.102:0/235879202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a681a07c0 0x7f3a681a0ba0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3a58005950 tx=0x7f3a58005780 comp rx=0 tx=0).stop 2024-04-26T20:24:56.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.637+0000 7f3a6ce00700 1 -- 172.21.15.102:0/235879202 shutdown_connections 2024-04-26T20:24:56.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.637+0000 7f3a6ce00700 1 --2- 172.21.15.102:0/235879202 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3a4c0396b0 0x7f3a4c03bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:56.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.637+0000 7f3a6ce00700 1 --2- 172.21.15.102:0/235879202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a681a07c0 0x7f3a681a0ba0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:56.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.637+0000 7f3a6ce00700 1 -- 172.21.15.102:0/235879202 >> 172.21.15.102:0/235879202 conn(0x7f3a680776c0 msgr2=0x7f3a680ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:56.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.638+0000 7f3a6ce00700 1 -- 172.21.15.102:0/235879202 shutdown_connections 2024-04-26T20:24:56.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:56.638+0000 7f3a6ce00700 1 -- 172.21.15.102:0/235879202 wait complete. 2024-04-26T20:24:57.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T20:24:57.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:57 smithi102 ceph-mon[28094]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:24:57.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:57 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:57.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:57 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:24:57.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:57 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:57.851 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:24:58.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:58 smithi102 ceph-mon[28094]: Added host smithi112 2024-04-26T20:24:58.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:58 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:59.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:59 smithi102 ceph-mon[28094]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:24:59.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:24:59 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:24:59.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.853+0000 7ff5ad600700 1 -- 172.21.15.102:0/4029753789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8103070 msgr2=0x7ff5a8103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:24:59.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.853+0000 7ff5ad600700 1 --2- 172.21.15.102:0/4029753789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8103070 0x7ff5a8103450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff598009910 tx=0x7ff598009c20 comp rx=0 tx=0).stop 2024-04-26T20:24:59.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.853+0000 7ff5ad600700 1 -- 172.21.15.102:0/4029753789 shutdown_connections 2024-04-26T20:24:59.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.853+0000 7ff5ad600700 1 --2- 172.21.15.102:0/4029753789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8103070 0x7ff5a8103450 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:24:59.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.853+0000 7ff5ad600700 1 -- 172.21.15.102:0/4029753789 >> 172.21.15.102:0/4029753789 conn(0x7ff5a80fe940 msgr2=0x7ff5a8100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:24:59.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.853+0000 7ff5ad600700 1 -- 172.21.15.102:0/4029753789 shutdown_connections 2024-04-26T20:24:59.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.853+0000 7ff5ad600700 1 -- 172.21.15.102:0/4029753789 wait complete. 2024-04-26T20:24:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.854+0000 7ff5ad600700 1 Processor -- start 2024-04-26T20:24:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.854+0000 7ff5ad600700 1 -- start start 2024-04-26T20:24:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.854+0000 7ff5ad600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8103070 0x7ff5a81a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.854+0000 7ff5ad600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a810d2f0 con 0x7ff5a8103070 2024-04-26T20:24:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.855+0000 7ff5a6a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8103070 0x7ff5a81a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.855+0000 7ff5a6a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8103070 0x7ff5a81a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40998/0 (socket says 172.21.15.102:40998) 2024-04-26T20:24:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.855+0000 7ff5a6a00700 1 -- 172.21.15.102:0/2627542071 learned_addr learned my addr 172.21.15.102:0/2627542071 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:24:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.855+0000 7ff5a6a00700 1 -- 172.21.15.102:0/2627542071 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a81a0bf0 con 0x7ff5a8103070 2024-04-26T20:24:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.855+0000 7ff5a6a00700 1 --2- 172.21.15.102:0/2627542071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8103070 0x7ff5a81a06b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff598006010 tx=0x7ff598005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.856+0000 7ff59f400700 1 -- 172.21.15.102:0/2627542071 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff598026070 con 0x7ff5a8103070 2024-04-26T20:24:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.856+0000 7ff59f400700 1 -- 172.21.15.102:0/2627542071 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff59800bba0 con 0x7ff5a8103070 2024-04-26T20:24:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.856+0000 7ff59f400700 1 -- 172.21.15.102:0/2627542071 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff598003910 con 0x7ff5a8103070 2024-04-26T20:24:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.856+0000 7ff5ad600700 1 -- 172.21.15.102:0/2627542071 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a81a0ee0 con 0x7ff5a8103070 2024-04-26T20:24:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.856+0000 7ff5ad600700 1 -- 172.21.15.102:0/2627542071 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a819a870 con 0x7ff5a8103070 2024-04-26T20:24:59.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.857+0000 7ff59f400700 1 -- 172.21.15.102:0/2627542071 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7ff598029070 con 0x7ff5a8103070 2024-04-26T20:24:59.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.857+0000 7ff5ad600700 1 -- 172.21.15.102:0/2627542071 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a8104550 con 0x7ff5a8103070 2024-04-26T20:24:59.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.857+0000 7ff59f400700 1 --2- 172.21.15.102:0/2627542071 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff58c0396b0 0x7ff58c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:24:59.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.857+0000 7ff59f400700 1 -- 172.21.15.102:0/2627542071 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff59801e070 con 0x7ff5a8103070 2024-04-26T20:24:59.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.858+0000 7ff5a6000700 1 --2- 172.21.15.102:0/2627542071 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff58c0396b0 0x7ff58c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:24:59.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.858+0000 7ff5a6000700 1 --2- 172.21.15.102:0/2627542071 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff58c0396b0 0x7ff58c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff594006fd0 tx=0x7ff594006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:24:59.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:24:59.863+0000 7ff59f400700 1 -- 172.21.15.102:0/2627542071 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff598013070 con 0x7ff5a8103070 2024-04-26T20:25:00.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.098+0000 7ff5ad600700 1 -- 172.21.15.102:0/2627542071 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff5a819bda0 con 0x7ff58c0396b0 2024-04-26T20:25:00.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.100+0000 7ff59f400700 1 -- 172.21.15.102:0/2627542071 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7ff5a819bda0 con 0x7ff58c0396b0 2024-04-26T20:25:00.101 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:25:00.101 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}, {"addr": "172.21.15.112", "hostname": "smithi112", "labels": [], "status": ""}] 2024-04-26T20:25:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.103+0000 7ff5ad600700 1 -- 172.21.15.102:0/2627542071 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff58c0396b0 msgr2=0x7ff58c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:00.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.103+0000 7ff5ad600700 1 --2- 172.21.15.102:0/2627542071 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff58c0396b0 0x7ff58c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff594006fd0 tx=0x7ff594006e40 comp rx=0 tx=0).stop 2024-04-26T20:25:00.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.103+0000 7ff5ad600700 1 -- 172.21.15.102:0/2627542071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8103070 msgr2=0x7ff5a81a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:00.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.103+0000 7ff5ad600700 1 --2- 172.21.15.102:0/2627542071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8103070 0x7ff5a81a06b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff598006010 tx=0x7ff598005780 comp rx=0 tx=0).stop 2024-04-26T20:25:00.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.103+0000 7ff5ad600700 1 -- 172.21.15.102:0/2627542071 shutdown_connections 2024-04-26T20:25:00.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.103+0000 7ff5ad600700 1 --2- 172.21.15.102:0/2627542071 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff58c0396b0 0x7ff58c03bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:00.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.103+0000 7ff5ad600700 1 --2- 172.21.15.102:0/2627542071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8103070 0x7ff5a81a06b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:00.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.103+0000 7ff5ad600700 1 -- 172.21.15.102:0/2627542071 >> 172.21.15.102:0/2627542071 conn(0x7ff5a80fe940 msgr2=0x7ff5a8075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:00.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.103+0000 7ff5ad600700 1 -- 172.21.15.102:0/2627542071 shutdown_connections 2024-04-26T20:25:00.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:00.103+0000 7ff5ad600700 1 -- 172.21.15.102:0/2627542071 wait complete. 2024-04-26T20:25:00.859 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi130 2024-04-26T20:25:00.859 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:25:00.859 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.conf 2024-04-26T20:25:00.878 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:25:00.878 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:25:00.935 INFO:tasks.cephadm:Adding host smithi130 to orchestrator... 2024-04-26T20:25:00.936 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch host add smithi130 2024-04-26T20:25:01.376 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:25:01.720 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:01 smithi102 ceph-mon[28094]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:25:01.720 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:01 smithi102 ceph-mon[28094]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:03.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.277+0000 7f7734400700 1 -- 172.21.15.102:0/4021678404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f772c103060 msgr2=0x7f772c103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:03.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.277+0000 7f7734400700 1 --2- 172.21.15.102:0/4021678404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f772c103060 0x7f772c103440 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7720009910 tx=0x7f7720009c20 comp rx=0 tx=0).stop 2024-04-26T20:25:03.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.277+0000 7f7734400700 1 -- 172.21.15.102:0/4021678404 shutdown_connections 2024-04-26T20:25:03.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.277+0000 7f7734400700 1 --2- 172.21.15.102:0/4021678404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f772c103060 0x7f772c103440 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:03.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.277+0000 7f7734400700 1 -- 172.21.15.102:0/4021678404 >> 172.21.15.102:0/4021678404 conn(0x7f772c0fe910 msgr2=0x7f772c100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:03.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.277+0000 7f7734400700 1 -- 172.21.15.102:0/4021678404 shutdown_connections 2024-04-26T20:25:03.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.277+0000 7f7734400700 1 -- 172.21.15.102:0/4021678404 wait complete. 2024-04-26T20:25:03.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.278+0000 7f7734400700 1 Processor -- start 2024-04-26T20:25:03.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.279+0000 7f7734400700 1 -- start start 2024-04-26T20:25:03.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.279+0000 7f7734400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f772c103060 0x7f772c1a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:03.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.279+0000 7f7734400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f772c076c40 con 0x7f772c103060 2024-04-26T20:25:03.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.279+0000 7f7731a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f772c103060 0x7f772c1a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:03.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.279+0000 7f7731a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f772c103060 0x7f772c1a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41016/0 (socket says 172.21.15.102:41016) 2024-04-26T20:25:03.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.279+0000 7f7731a00700 1 -- 172.21.15.102:0/2473822019 learned_addr learned my addr 172.21.15.102:0/2473822019 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:25:03.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.280+0000 7f7731a00700 1 -- 172.21.15.102:0/2473822019 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772c1a0c00 con 0x7f772c103060 2024-04-26T20:25:03.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.280+0000 7f7731a00700 1 --2- 172.21.15.102:0/2473822019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f772c103060 0x7f772c1a06c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7720006010 tx=0x7f7720005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:03.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.280+0000 7f772a000700 1 -- 172.21.15.102:0/2473822019 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7720026070 con 0x7f772c103060 2024-04-26T20:25:03.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.280+0000 7f772a000700 1 -- 172.21.15.102:0/2473822019 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f772000bba0 con 0x7f772c103060 2024-04-26T20:25:03.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.280+0000 7f7734400700 1 -- 172.21.15.102:0/2473822019 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f772c1a0ef0 con 0x7f772c103060 2024-04-26T20:25:03.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.280+0000 7f772a000700 1 -- 172.21.15.102:0/2473822019 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7720003910 con 0x7f772c103060 2024-04-26T20:25:03.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.281+0000 7f7734400700 1 -- 172.21.15.102:0/2473822019 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f772c19a880 con 0x7f772c103060 2024-04-26T20:25:03.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.282+0000 7f7734400700 1 -- 172.21.15.102:0/2473822019 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f772c104540 con 0x7f772c103060 2024-04-26T20:25:03.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.282+0000 7f772a000700 1 -- 172.21.15.102:0/2473822019 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f7720029070 con 0x7f772c103060 2024-04-26T20:25:03.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.282+0000 7f772a000700 1 --2- 172.21.15.102:0/2473822019 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77140396b0 0x7f771403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:03.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.282+0000 7f772a000700 1 -- 172.21.15.102:0/2473822019 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f772001e070 con 0x7f772c103060 2024-04-26T20:25:03.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.282+0000 7f7731000700 1 --2- 172.21.15.102:0/2473822019 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77140396b0 0x7f771403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:03.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.283+0000 7f7731000700 1 --2- 172.21.15.102:0/2473822019 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77140396b0 0x7f771403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7718006fd0 tx=0x7f7718006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:03.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.288+0000 7f772a000700 1 -- 172.21.15.102:0/2473822019 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7720013070 con 0x7f772c103060 2024-04-26T20:25:03.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:03.521+0000 7f7734400700 1 -- 172.21.15.102:0/2473822019 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}) -- 0x7f772c19bdb0 con 0x7f77140396b0 2024-04-26T20:25:03.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:03 smithi102 ceph-mon[28094]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:03.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:03.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:03.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:03.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:03.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:25:03.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:03.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:04.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:04 smithi102 ceph-mon[28094]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-26T20:25:04.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:04 smithi102 ceph-mon[28094]: Updating smithi112:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:04.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:04 smithi102 ceph-mon[28094]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:25:04.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:04 smithi102 ceph-mon[28094]: Updating smithi112:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T20:25:04.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:04 smithi102 ceph-mon[28094]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:04.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:04.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:04.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:05.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:05 smithi102 ceph-mon[28094]: Deploying cephadm binary to smithi130 2024-04-26T20:25:05.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:05 smithi102 ceph-mon[28094]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:07.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:07 smithi102 ceph-mon[28094]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:08.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.107+0000 7f772a000700 1 -- 172.21.15.102:0/2473822019 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f772c19bdb0 con 0x7f77140396b0 2024-04-26T20:25:08.108 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi130' with addr '172.21.15.130' 2024-04-26T20:25:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.110+0000 7f7734400700 1 -- 172.21.15.102:0/2473822019 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77140396b0 msgr2=0x7f771403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.110+0000 7f7734400700 1 --2- 172.21.15.102:0/2473822019 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77140396b0 0x7f771403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7718006fd0 tx=0x7f7718006e40 comp rx=0 tx=0).stop 2024-04-26T20:25:08.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.110+0000 7f7734400700 1 -- 172.21.15.102:0/2473822019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f772c103060 msgr2=0x7f772c1a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:08.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.110+0000 7f7734400700 1 --2- 172.21.15.102:0/2473822019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f772c103060 0x7f772c1a06c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7720006010 tx=0x7f7720005780 comp rx=0 tx=0).stop 2024-04-26T20:25:08.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.111+0000 7f7734400700 1 -- 172.21.15.102:0/2473822019 shutdown_connections 2024-04-26T20:25:08.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.111+0000 7f7734400700 1 --2- 172.21.15.102:0/2473822019 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77140396b0 0x7f771403bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:08.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.111+0000 7f7734400700 1 --2- 172.21.15.102:0/2473822019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f772c103060 0x7f772c1a06c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:08.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.111+0000 7f7734400700 1 -- 172.21.15.102:0/2473822019 >> 172.21.15.102:0/2473822019 conn(0x7f772c0fe910 msgr2=0x7f772c0ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:08.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.111+0000 7f7734400700 1 -- 172.21.15.102:0/2473822019 shutdown_connections 2024-04-26T20:25:08.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:08.111+0000 7f7734400700 1 -- 172.21.15.102:0/2473822019 wait complete. 2024-04-26T20:25:08.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T20:25:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:25:09.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:09.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:09 smithi102 ceph-mon[28094]: Added host smithi130 2024-04-26T20:25:09.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:09.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:09 smithi102 ceph-mon[28094]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:09.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:10.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:10 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.392+0000 7f1856a00700 1 -- 172.21.15.102:0/4220448054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185010f340 msgr2=0x7f185010f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.392+0000 7f1856a00700 1 --2- 172.21.15.102:0/4220448054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185010f340 0x7f185010f720 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1844009910 tx=0x7f1844009c20 comp rx=0 tx=0).stop 2024-04-26T20:25:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.392+0000 7f1856a00700 1 -- 172.21.15.102:0/4220448054 shutdown_connections 2024-04-26T20:25:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.392+0000 7f1856a00700 1 --2- 172.21.15.102:0/4220448054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185010f340 0x7f185010f720 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.392+0000 7f1856a00700 1 -- 172.21.15.102:0/4220448054 >> 172.21.15.102:0/4220448054 conn(0x7f1850078a20 msgr2=0x7f185007ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.393+0000 7f1856a00700 1 -- 172.21.15.102:0/4220448054 shutdown_connections 2024-04-26T20:25:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.393+0000 7f1856a00700 1 -- 172.21.15.102:0/4220448054 wait complete. 2024-04-26T20:25:11.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.394+0000 7f1856a00700 1 Processor -- start 2024-04-26T20:25:11.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.394+0000 7f1856a00700 1 -- start start 2024-04-26T20:25:11.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.395+0000 7f1856a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185010f340 0x7f18501a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:11.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.395+0000 7f1856a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1850113740 con 0x7f185010f340 2024-04-26T20:25:11.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.395+0000 7f1855600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185010f340 0x7f18501a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:11.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.395+0000 7f1855600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185010f340 0x7f18501a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50208/0 (socket says 172.21.15.102:50208) 2024-04-26T20:25:11.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.395+0000 7f1855600700 1 -- 172.21.15.102:0/3495351127 learned_addr learned my addr 172.21.15.102:0/3495351127 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:25:11.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.395+0000 7f1855600700 1 -- 172.21.15.102:0/3495351127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18501a4df0 con 0x7f185010f340 2024-04-26T20:25:11.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.396+0000 7f1855600700 1 --2- 172.21.15.102:0/3495351127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185010f340 0x7f18501a48b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1844006010 tx=0x7f1844005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:11.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.396+0000 7f184e000700 1 -- 172.21.15.102:0/3495351127 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1844026070 con 0x7f185010f340 2024-04-26T20:25:11.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.396+0000 7f184e000700 1 -- 172.21.15.102:0/3495351127 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f184400bba0 con 0x7f185010f340 2024-04-26T20:25:11.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.396+0000 7f1856a00700 1 -- 172.21.15.102:0/3495351127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18501a50e0 con 0x7f185010f340 2024-04-26T20:25:11.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.396+0000 7f184e000700 1 -- 172.21.15.102:0/3495351127 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1844003930 con 0x7f185010f340 2024-04-26T20:25:11.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.396+0000 7f1856a00700 1 -- 172.21.15.102:0/3495351127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f185019ea70 con 0x7f185010f340 2024-04-26T20:25:11.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.397+0000 7f1856a00700 1 -- 172.21.15.102:0/3495351127 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185010a980 con 0x7f185010f340 2024-04-26T20:25:11.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.397+0000 7f184e000700 1 -- 172.21.15.102:0/3495351127 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f1844029070 con 0x7f185010f340 2024-04-26T20:25:11.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.398+0000 7f184e000700 1 --2- 172.21.15.102:0/3495351127 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18380396b0 0x7f183803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:11.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.398+0000 7f184e000700 1 -- 172.21.15.102:0/3495351127 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f184401e070 con 0x7f185010f340 2024-04-26T20:25:11.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.398+0000 7f1854c00700 1 --2- 172.21.15.102:0/3495351127 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18380396b0 0x7f183803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:11.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.399+0000 7f1854c00700 1 --2- 172.21.15.102:0/3495351127 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18380396b0 0x7f183803bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f183c006fd0 tx=0x7f183c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:11.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.404+0000 7f184e000700 1 -- 172.21.15.102:0/3495351127 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1844013070 con 0x7f185010f340 2024-04-26T20:25:11.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.634+0000 7f1856a00700 1 -- 172.21.15.102:0/3495351127 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f18501a00b0 con 0x7f18380396b0 2024-04-26T20:25:11.636 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:11 smithi102 ceph-mon[28094]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:11.636 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:11.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.636+0000 7f184e000700 1 -- 172.21.15.102:0/3495351127 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f18501a00b0 con 0x7f18380396b0 2024-04-26T20:25:11.638 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:25:11.638 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}, {"addr": "172.21.15.112", "hostname": "smithi112", "labels": [], "status": ""}, {"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}] 2024-04-26T20:25:11.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.640+0000 7f1856a00700 1 -- 172.21.15.102:0/3495351127 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18380396b0 msgr2=0x7f183803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:11.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.640+0000 7f1856a00700 1 --2- 172.21.15.102:0/3495351127 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18380396b0 0x7f183803bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f183c006fd0 tx=0x7f183c006e40 comp rx=0 tx=0).stop 2024-04-26T20:25:11.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.640+0000 7f1856a00700 1 -- 172.21.15.102:0/3495351127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185010f340 msgr2=0x7f18501a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:11.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.640+0000 7f1856a00700 1 --2- 172.21.15.102:0/3495351127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185010f340 0x7f18501a48b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1844006010 tx=0x7f1844005780 comp rx=0 tx=0).stop 2024-04-26T20:25:11.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.640+0000 7f1856a00700 1 -- 172.21.15.102:0/3495351127 shutdown_connections 2024-04-26T20:25:11.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.640+0000 7f1856a00700 1 --2- 172.21.15.102:0/3495351127 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18380396b0 0x7f183803bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:11.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.640+0000 7f1856a00700 1 --2- 172.21.15.102:0/3495351127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185010f340 0x7f18501a48b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:11.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.640+0000 7f1856a00700 1 -- 172.21.15.102:0/3495351127 >> 172.21.15.102:0/3495351127 conn(0x7f1850078a20 msgr2=0x7f185007a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:11.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.640+0000 7f1856a00700 1 -- 172.21.15.102:0/3495351127 shutdown_connections 2024-04-26T20:25:11.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:11.640+0000 7f1856a00700 1 -- 172.21.15.102:0/3495351127 wait complete. 2024-04-26T20:25:12.364 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T20:25:12.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T20:25:12.781 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:25:13.867 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:13 smithi102 ceph-mon[28094]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:25:13.867 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:13 smithi102 ceph-mon[28094]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.759+0000 7f2688000700 1 -- 172.21.15.102:0/3772202700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2680104020 msgr2=0x7f2680104400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.759+0000 7f2688000700 1 --2- 172.21.15.102:0/3772202700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2680104020 0x7f2680104400 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f267400b0a0 tx=0x7f267400b3b0 comp rx=0 tx=0).stop 2024-04-26T20:25:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.759+0000 7f2688000700 1 -- 172.21.15.102:0/3772202700 shutdown_connections 2024-04-26T20:25:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.759+0000 7f2688000700 1 --2- 172.21.15.102:0/3772202700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2680104020 0x7f2680104400 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.759+0000 7f2688000700 1 -- 172.21.15.102:0/3772202700 >> 172.21.15.102:0/3772202700 conn(0x7f26800789f0 msgr2=0x7f26800ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.759+0000 7f2688000700 1 -- 172.21.15.102:0/3772202700 shutdown_connections 2024-04-26T20:25:14.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.759+0000 7f2688000700 1 -- 172.21.15.102:0/3772202700 wait complete. 2024-04-26T20:25:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.760+0000 7f2688000700 1 Processor -- start 2024-04-26T20:25:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.760+0000 7f2688000700 1 -- start start 2024-04-26T20:25:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.761+0000 7f2688000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2680104020 0x7f2680101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.761+0000 7f2688000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f268010d350 con 0x7f2680104020 2024-04-26T20:25:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.761+0000 7f2685600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2680104020 0x7f2680101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:14.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.761+0000 7f2685600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2680104020 0x7f2680101c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50226/0 (socket says 172.21.15.102:50226) 2024-04-26T20:25:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.761+0000 7f2685600700 1 -- 172.21.15.102:0/2687915732 learned_addr learned my addr 172.21.15.102:0/2687915732 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:25:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.762+0000 7f2685600700 1 -- 172.21.15.102:0/2687915732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2680103b70 con 0x7f2680104020 2024-04-26T20:25:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.762+0000 7f2685600700 1 --2- 172.21.15.102:0/2687915732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2680104020 0x7f2680101c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f267400d010 tx=0x7f2674009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.762+0000 7f267e000700 1 -- 172.21.15.102:0/2687915732 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2674021020 con 0x7f2680104020 2024-04-26T20:25:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.762+0000 7f267e000700 1 -- 172.21.15.102:0/2687915732 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2674004000 con 0x7f2680104020 2024-04-26T20:25:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.762+0000 7f2688000700 1 -- 172.21.15.102:0/2687915732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2680102220 con 0x7f2680104020 2024-04-26T20:25:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.762+0000 7f267e000700 1 -- 172.21.15.102:0/2687915732 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2674017040 con 0x7f2680104020 2024-04-26T20:25:14.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.762+0000 7f2688000700 1 -- 172.21.15.102:0/2687915732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26801026e0 con 0x7f2680104020 2024-04-26T20:25:14.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.763+0000 7f2688000700 1 -- 172.21.15.102:0/2687915732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f268006b0d0 con 0x7f2680104020 2024-04-26T20:25:14.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.763+0000 7f267e000700 1 -- 172.21.15.102:0/2687915732 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f2674024020 con 0x7f2680104020 2024-04-26T20:25:14.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.764+0000 7f267e000700 1 --2- 172.21.15.102:0/2687915732 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f26680396f0 0x7f266803bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:14.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.764+0000 7f267e000700 1 -- 172.21.15.102:0/2687915732 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f267400f050 con 0x7f2680104020 2024-04-26T20:25:14.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.764+0000 7f2684c00700 1 --2- 172.21.15.102:0/2687915732 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f26680396f0 0x7f266803bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:14.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.765+0000 7f2684c00700 1 --2- 172.21.15.102:0/2687915732 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f26680396f0 0x7f266803bbb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f266c006fd0 tx=0x7f266c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:14.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:14.772+0000 7f267e000700 1 -- 172.21.15.102:0/2687915732 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f267402b020 con 0x7f2680104020 2024-04-26T20:25:15.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.008+0000 7f2688000700 1 -- 172.21.15.102:0/2687915732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f26801031b0 con 0x7f2680104020 2024-04-26T20:25:15.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.120+0000 7f267e000700 1 -- 172.21.15.102:0/2687915732 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f2674011390 con 0x7f2680104020 2024-04-26T20:25:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.125+0000 7f2688000700 1 -- 172.21.15.102:0/2687915732 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f26680396f0 msgr2=0x7f266803bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.125+0000 7f2688000700 1 --2- 172.21.15.102:0/2687915732 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f26680396f0 0x7f266803bbb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f266c006fd0 tx=0x7f266c006e40 comp rx=0 tx=0).stop 2024-04-26T20:25:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.125+0000 7f2688000700 1 -- 172.21.15.102:0/2687915732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2680104020 msgr2=0x7f2680101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.125+0000 7f2688000700 1 --2- 172.21.15.102:0/2687915732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2680104020 0x7f2680101c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f267400d010 tx=0x7f2674009790 comp rx=0 tx=0).stop 2024-04-26T20:25:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.126+0000 7f2688000700 1 -- 172.21.15.102:0/2687915732 shutdown_connections 2024-04-26T20:25:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.126+0000 7f2688000700 1 --2- 172.21.15.102:0/2687915732 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f26680396f0 0x7f266803bbb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.126+0000 7f2688000700 1 --2- 172.21.15.102:0/2687915732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2680104020 0x7f2680101c80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.126+0000 7f2688000700 1 -- 172.21.15.102:0/2687915732 >> 172.21.15.102:0/2687915732 conn(0x7f26800789f0 msgr2=0x7f2680104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.126+0000 7f2688000700 1 -- 172.21.15.102:0/2687915732 shutdown_connections 2024-04-26T20:25:15.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:15.126+0000 7f2688000700 1 -- 172.21.15.102:0/2687915732 wait complete. 2024-04-26T20:25:15.128 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2024-04-26T20:25:15.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:15.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:15.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:15.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:15.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:25:15.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:15.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:15.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-26T20:25:15.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:15.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: Updating smithi130:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:15.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:15 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2687915732' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T20:25:15.855 INFO:tasks.cephadm:Adding mon.a on smithi102 2024-04-26T20:25:15.855 INFO:tasks.cephadm:Adding mon.b on smithi112 2024-04-26T20:25:15.855 INFO:tasks.cephadm:Adding mon.c on smithi130 2024-04-26T20:25:15.855 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi102:172.21.15.102=a;smithi112:172.21.15.112=b;smithi130:172.21.15.130=c' 2024-04-26T20:25:16.081 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:16.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:16 smithi102 ceph-mon[28094]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:25:16.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:16 smithi102 ceph-mon[28094]: Updating smithi130:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T20:25:16.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:16 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2687915732' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T20:25:16.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:16 smithi102 ceph-mon[28094]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:25:16.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:16 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:16.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:16 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:16.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:16 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:17.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:17 smithi102 ceph-mon[28094]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:17.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.550+0000 7f775559a700 1 -- 172.21.15.130:0/1344459087 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7748005460 con 0x7f7758101f70 2024-04-26T20:25:17.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.551+0000 7f775cdfb700 1 -- 172.21.15.130:0/1344459087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7758101f70 msgr2=0x7f7758102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:17.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.551+0000 7f775cdfb700 1 --2- 172.21.15.130:0/1344459087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7758101f70 0x7f7758102350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7748009910 tx=0x7f7748009c20 comp rx=0 tx=0).stop 2024-04-26T20:25:17.556 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.551+0000 7f775cdfb700 1 -- 172.21.15.130:0/1344459087 shutdown_connections 2024-04-26T20:25:17.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.551+0000 7f775cdfb700 1 --2- 172.21.15.130:0/1344459087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7758101f70 0x7f7758102350 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:17.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.551+0000 7f775cdfb700 1 -- 172.21.15.130:0/1344459087 >> 172.21.15.130:0/1344459087 conn(0x7f77580fd820 msgr2=0x7f77580ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:17.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.551+0000 7f775cdfb700 1 -- 172.21.15.130:0/1344459087 shutdown_connections 2024-04-26T20:25:17.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.551+0000 7f775cdfb700 1 -- 172.21.15.130:0/1344459087 wait complete. 2024-04-26T20:25:17.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.552+0000 7f775cdfb700 1 Processor -- start 2024-04-26T20:25:17.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.552+0000 7f775cdfb700 1 -- start start 2024-04-26T20:25:17.557 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.552+0000 7f775cdfb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7758101f70 0x7f77581a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:17.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.552+0000 7f775cdfb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f775810b0e0 con 0x7f7758101f70 2024-04-26T20:25:17.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.553+0000 7f775659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7758101f70 0x7f77581a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:17.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.553+0000 7f775659c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7758101f70 0x7f77581a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.130:37038/0 (socket says 172.21.15.130:37038) 2024-04-26T20:25:17.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.553+0000 7f775659c700 1 -- 172.21.15.130:0/2312932835 learned_addr learned my addr 172.21.15.130:0/2312932835 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-26T20:25:17.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.553+0000 7f775659c700 1 -- 172.21.15.130:0/2312932835 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77581a0c10 con 0x7f7758101f70 2024-04-26T20:25:17.558 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.554+0000 7f775659c700 1 --2- 172.21.15.130:0/2312932835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7758101f70 0x7f77581a06d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f77480056d0 tx=0x7f77480057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:17.559 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.554+0000 7f77477fe700 1 -- 172.21.15.130:0/2312932835 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7748026070 con 0x7f7758101f70 2024-04-26T20:25:17.559 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.554+0000 7f775cdfb700 1 -- 172.21.15.130:0/2312932835 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77581a0f00 con 0x7f7758101f70 2024-04-26T20:25:17.559 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.554+0000 7f77477fe700 1 -- 172.21.15.130:0/2312932835 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f774800bba0 con 0x7f7758101f70 2024-04-26T20:25:17.559 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.554+0000 7f77477fe700 1 -- 172.21.15.130:0/2312932835 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7748003920 con 0x7f7758101f70 2024-04-26T20:25:17.559 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.554+0000 7f775cdfb700 1 -- 172.21.15.130:0/2312932835 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f775819a890 con 0x7f7758101f70 2024-04-26T20:25:17.559 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.555+0000 7f775cdfb700 1 -- 172.21.15.130:0/2312932835 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7758103450 con 0x7f7758101f70 2024-04-26T20:25:17.561 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.556+0000 7f77477fe700 1 -- 172.21.15.130:0/2312932835 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f7748029070 con 0x7f7758101f70 2024-04-26T20:25:17.561 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.556+0000 7f77477fe700 1 --2- 172.21.15.130:0/2312932835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7740039650 0x7f774003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:17.561 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.556+0000 7f77477fe700 1 -- 172.21.15.130:0/2312932835 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f774801e070 con 0x7f7758101f70 2024-04-26T20:25:17.561 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.556+0000 7f7755d9b700 1 --2- 172.21.15.130:0/2312932835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7740039650 0x7f774003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:17.562 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.557+0000 7f7755d9b700 1 --2- 172.21.15.130:0/2312932835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7740039650 0x7f774003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f774c006fd0 tx=0x7f774c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:17.572 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.571+0000 7f77477fe700 1 -- 172.21.15.130:0/2312932835 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7748013070 con 0x7f7758101f70 2024-04-26T20:25:17.784 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.782+0000 7f775cdfb700 1 -- 172.21.15.130:0/2312932835 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi112:172.21.15.112=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}) -- 0x7f775819be30 con 0x7f7740039650 2024-04-26T20:25:17.798 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.797+0000 7f77477fe700 1 -- 172.21.15.130:0/2312932835 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f775819be30 con 0x7f7740039650 2024-04-26T20:25:17.798 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mon update... 2024-04-26T20:25:17.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.799+0000 7f775cdfb700 1 -- 172.21.15.130:0/2312932835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7740039650 msgr2=0x7f774003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:17.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.799+0000 7f775cdfb700 1 --2- 172.21.15.130:0/2312932835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7740039650 0x7f774003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f774c006fd0 tx=0x7f774c006e40 comp rx=0 tx=0).stop 2024-04-26T20:25:17.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.799+0000 7f775cdfb700 1 -- 172.21.15.130:0/2312932835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7758101f70 msgr2=0x7f77581a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:17.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.799+0000 7f775cdfb700 1 --2- 172.21.15.130:0/2312932835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7758101f70 0x7f77581a06d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f77480056d0 tx=0x7f77480057b0 comp rx=0 tx=0).stop 2024-04-26T20:25:17.800 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.799+0000 7f775cdfb700 1 -- 172.21.15.130:0/2312932835 shutdown_connections 2024-04-26T20:25:17.801 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.799+0000 7f775cdfb700 1 --2- 172.21.15.130:0/2312932835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7740039650 0x7f774003bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:17.801 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.799+0000 7f775cdfb700 1 --2- 172.21.15.130:0/2312932835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7758101f70 0x7f77581a06d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:17.801 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.799+0000 7f775cdfb700 1 -- 172.21.15.130:0/2312932835 >> 172.21.15.130:0/2312932835 conn(0x7f77580fd820 msgr2=0x7f7758103940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:17.801 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.799+0000 7f775cdfb700 1 -- 172.21.15.130:0/2312932835 shutdown_connections 2024-04-26T20:25:17.801 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:17.799+0000 7f775cdfb700 1 -- 172.21.15.130:0/2312932835 wait complete. 2024-04-26T20:25:18.367 DEBUG:teuthology.orchestra.run.smithi112:mon.b> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T20:25:18.370 DEBUG:teuthology.orchestra.run.smithi130:mon.c> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T20:25:18.373 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T20:25:18.373 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi112:172.21.15.112=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi112:172.21.15.112=b;smithi130:172.21.15.130=c;count:3 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: Deploying daemon mon.c on smithi130 2024-04-26T20:25:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:18 smithi102 ceph-mon[28094]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:19.408 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:21.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:21 smithi102 ceph-mon[28094]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.026 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.024+0000 7f5bf338f700 1 -- 172.21.15.130:0/1400697995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec101670 msgr2=0x7f5bec101a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:22.027 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.024+0000 7f5bf338f700 1 --2- 172.21.15.130:0/1400697995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec101670 0x7f5bec101a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009910 tx=0x7f5be8009c20 comp rx=0 tx=0).stop 2024-04-26T20:25:22.027 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.024+0000 7f5bf338f700 1 -- 172.21.15.130:0/1400697995 shutdown_connections 2024-04-26T20:25:22.027 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.024+0000 7f5bf338f700 1 --2- 172.21.15.130:0/1400697995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec101670 0x7f5bec101a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:22.027 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.024+0000 7f5bf338f700 1 -- 172.21.15.130:0/1400697995 >> 172.21.15.130:0/1400697995 conn(0x7f5bec078410 msgr2=0x7f5bec078820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:22.027 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.024+0000 7f5bf338f700 1 -- 172.21.15.130:0/1400697995 shutdown_connections 2024-04-26T20:25:22.027 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.025+0000 7f5bf338f700 1 -- 172.21.15.130:0/1400697995 wait complete. 2024-04-26T20:25:22.027 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.025+0000 7f5bf338f700 1 Processor -- start 2024-04-26T20:25:22.027 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.025+0000 7f5bf338f700 1 -- start start 2024-04-26T20:25:22.027 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.026+0000 7f5bf338f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec101670 0x7f5bec19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:22.027 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.026+0000 7f5bf338f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5bec10d300 con 0x7f5bec101670 2024-04-26T20:25:22.029 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.026+0000 7f5bf112b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec101670 0x7f5bec19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:22.029 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.026+0000 7f5bf112b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec101670 0x7f5bec19e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.130:37060/0 (socket says 172.21.15.130:37060) 2024-04-26T20:25:22.029 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.026+0000 7f5bf112b700 1 -- 172.21.15.130:0/1688293865 learned_addr learned my addr 172.21.15.130:0/1688293865 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-26T20:25:22.029 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.027+0000 7f5bf112b700 1 -- 172.21.15.130:0/1688293865 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bec19eb60 con 0x7f5bec101670 2024-04-26T20:25:22.029 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.027+0000 7f5bf112b700 1 --2- 172.21.15.130:0/1688293865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec101670 0x7f5bec19e620 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5be80056d0 tx=0x7f5be80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:22.029 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.028+0000 7f5be27fc700 1 -- 172.21.15.130:0/1688293865 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5be8026070 con 0x7f5bec101670 2024-04-26T20:25:22.029 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.028+0000 7f5bf338f700 1 -- 172.21.15.130:0/1688293865 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bec19ee50 con 0x7f5bec101670 2024-04-26T20:25:22.029 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.028+0000 7f5bf338f700 1 -- 172.21.15.130:0/1688293865 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bec198780 con 0x7f5bec101670 2024-04-26T20:25:22.029 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.028+0000 7f5be27fc700 1 -- 172.21.15.130:0/1688293865 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5be800b6f0 con 0x7f5bec101670 2024-04-26T20:25:22.029 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.028+0000 7f5be27fc700 1 -- 172.21.15.130:0/1688293865 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5be8030060 con 0x7f5bec101670 2024-04-26T20:25:22.030 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.029+0000 7f5bf338f700 1 -- 172.21.15.130:0/1688293865 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bd0005330 con 0x7f5bec101670 2024-04-26T20:25:22.030 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.029+0000 7f5be27fc700 1 -- 172.21.15.130:0/1688293865 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f5be8029070 con 0x7f5bec101670 2024-04-26T20:25:22.030 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.029+0000 7f5be27fc700 1 --2- 172.21.15.130:0/1688293865 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5bd8039630 0x7f5bd803baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:22.030 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.030+0000 7f5be27fc700 1 -- 172.21.15.130:0/1688293865 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5be801e070 con 0x7f5bec101670 2024-04-26T20:25:22.031 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.030+0000 7f5bf092a700 1 --2- 172.21.15.130:0/1688293865 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5bd8039630 0x7f5bd803baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:22.032 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.031+0000 7f5bf092a700 1 --2- 172.21.15.130:0/1688293865 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5bd8039630 0x7f5bd803baf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc006fd0 tx=0x7f5bdc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:22.046 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.045+0000 7f5be27fc700 1 -- 172.21.15.130:0/1688293865 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5be8013070 con 0x7f5bec101670 2024-04-26T20:25:22.334 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.333+0000 7f5bf338f700 1 -- 172.21.15.130:0/1688293865 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5bd0005620 con 0x7f5bec101670 2024-04-26T20:25:22.335 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.334+0000 7f5be27fc700 1 -- 172.21.15.130:0/1688293865 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f5be800b8a0 con 0x7f5bec101670 2024-04-26T20:25:22.335 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:25:22.335 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":1,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","modified":"2024-04-26T20:23:15.907271Z","created":"2024-04-26T20:23:15.907271Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T20:25:22.337 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.336+0000 7f5bf338f700 1 -- 172.21.15.130:0/1688293865 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5bd8039630 msgr2=0x7f5bd803baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:22.337 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.336+0000 7f5bf338f700 1 --2- 172.21.15.130:0/1688293865 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5bd8039630 0x7f5bd803baf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc006fd0 tx=0x7f5bdc006e40 comp rx=0 tx=0).stop 2024-04-26T20:25:22.337 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.336+0000 7f5bf338f700 1 -- 172.21.15.130:0/1688293865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec101670 msgr2=0x7f5bec19e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:22.337 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.336+0000 7f5bf338f700 1 --2- 172.21.15.130:0/1688293865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec101670 0x7f5bec19e620 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5be80056d0 tx=0x7f5be80057b0 comp rx=0 tx=0).stop 2024-04-26T20:25:22.337 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.336+0000 7f5bf338f700 1 -- 172.21.15.130:0/1688293865 shutdown_connections 2024-04-26T20:25:22.337 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.336+0000 7f5bf338f700 1 --2- 172.21.15.130:0/1688293865 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5bd8039630 0x7f5bd803baf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:22.337 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.336+0000 7f5bf338f700 1 --2- 172.21.15.130:0/1688293865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec101670 0x7f5bec19e620 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:22.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.336+0000 7f5bf338f700 1 -- 172.21.15.130:0/1688293865 >> 172.21.15.130:0/1688293865 conn(0x7f5bec078410 msgr2=0x7f5bec104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:22.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.336+0000 7f5bf338f700 1 -- 172.21.15.130:0/1688293865 shutdown_connections 2024-04-26T20:25:22.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:22.336+0000 7f5bf338f700 1 -- 172.21.15.130:0/1688293865 wait complete. 2024-04-26T20:25:22.338 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 1 2024-04-26T20:25:22.402 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).mds e1 new map 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout: e1 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout: legacy client fscid: -1 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout: 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout: No filesystems configured 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T20:25:22.403 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Updating smithi102:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Deploying cephadm binary to smithi112 2024-04-26T20:25:22.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.405 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.405 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.405 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:22.405 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.405 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Added host smithi112 2024-04-26T20:25:22.405 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.405 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.405 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.405 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Updating smithi112:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:22.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Updating smithi112:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Deploying cephadm binary to smithi130 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Added host smithi130 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Updating smithi130:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2687915732' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T20:25:22.408 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Updating smithi130:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2687915732' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi112:172.21.15.112=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi112:172.21.15.112=b;smithi130:172.21.15.130=c;count:3 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:22.409 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:22.410 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:22.410 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: Deploying daemon mon.c on smithi130 2024-04-26T20:25:22.410 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.410 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:22.410 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 ceph-mon[32704]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T20:25:22.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 bash[32665]: a8d92a4102eaaec9d97e52aabe65f8b18647b21a56a20ab10ecf398d8dd41dc3 2024-04-26T20:25:22.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:22 smithi130 systemd[1]: Started Ceph mon.c for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:25:24.030 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T20:25:24.031 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T20:25:24.284 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: mon.a calling monitor election 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: mon.c calling monitor election 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: monmap epoch 2 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: last_changed 2024-04-26T20:25:22.408227+0000 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: created 2024-04-26T20:23:15.907271+0000 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: min_mon_release 19 (squid) 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: election_strategy: 1 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: fsmap 2024-04-26T20:25:27.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:25:27.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: mgrmap e13: x(active, since 59s) 2024-04-26T20:25:27.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: overall HEALTH_OK 2024-04-26T20:25:27.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:27.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:27.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: mon.a calling monitor election 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: mon.c calling monitor election 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: monmap epoch 2 2024-04-26T20:25:27.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: last_changed 2024-04-26T20:25:22.408227+0000 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: created 2024-04-26T20:23:15.907271+0000 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: min_mon_release 19 (squid) 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: election_strategy: 1 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: fsmap 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: mgrmap e13: x(active, since 59s) 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: overall HEALTH_OK 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:27.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:28.605 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.603+0000 7f764c0c8700 1 -- 172.21.15.130:0/2042192055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76300035c0 msgr2=0x7f7630005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:28.605 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.603+0000 7f764c0c8700 1 --2- 172.21.15.130:0/2042192055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76300035c0 0x7f7630005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7644101d40 tx=0x7f763400feb0 comp rx=0 tx=0).stop 2024-04-26T20:25:28.605 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.603+0000 7f764c0c8700 1 -- 172.21.15.130:0/2042192055 shutdown_connections 2024-04-26T20:25:28.605 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.603+0000 7f764c0c8700 1 --2- 172.21.15.130:0/2042192055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76300035c0 0x7f7630005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:28.605 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.603+0000 7f764c0c8700 1 -- 172.21.15.130:0/2042192055 >> 172.21.15.130:0/2042192055 conn(0x7f76440fd900 msgr2=0x7f76440ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:28.605 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.603+0000 7f764c0c8700 1 -- 172.21.15.130:0/2042192055 shutdown_connections 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.603+0000 7f764c0c8700 1 -- 172.21.15.130:0/2042192055 wait complete. 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f764c0c8700 1 Processor -- start 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f764c0c8700 1 -- start start 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f764c0c8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76300035c0 0x7f7644199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f764c0c8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7644199d60 0x7f764419a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f764c0c8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7644109d60 con 0x7f76300035c0 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f764c0c8700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7644109be0 con 0x7f7644199d60 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f7649663700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7644199d60 0x7f764419a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f7649663700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7644199d60 0x7f764419a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:58770/0 (socket says 172.21.15.130:58770) 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f7649663700 1 -- 172.21.15.130:0/2509936129 learned_addr learned my addr 172.21.15.130:0/2509936129 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f7649663700 1 -- 172.21.15.130:0/2509936129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7644199d60 msgr2=0x7f764419a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T20:25:28.606 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f7649663700 1 -- 172.21.15.130:0/2509936129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7644199d60 msgr2=0x7f764419a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T20:25:28.607 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f7649663700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7644199d60 0x7f764419a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T20:25:28.607 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f7649663700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7644199d60 0x7f764419a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:25:28.607 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.604+0000 7f7649e64700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76300035c0 0x7f7644199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:28.608 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.605+0000 7f7649e64700 1 -- 172.21.15.130:0/2509936129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7644199d60 msgr2=0x7f764419a1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:25:28.608 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.605+0000 7f7649e64700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7644199d60 0x7f764419a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:28.608 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.605+0000 7f7649e64700 1 -- 172.21.15.130:0/2509936129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76441a2890 con 0x7f76300035c0 2024-04-26T20:25:28.608 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.605+0000 7f7649e64700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76300035c0 0x7f7644199820 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7634010010 tx=0x7f76340116a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:28.608 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.606+0000 7f763affd700 1 -- 172.21.15.130:0/2509936129 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7634013960 con 0x7f76300035c0 2024-04-26T20:25:28.608 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.606+0000 7f764c0c8700 1 -- 172.21.15.130:0/2509936129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76441a2a60 con 0x7f76300035c0 2024-04-26T20:25:28.609 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.606+0000 7f764c0c8700 1 -- 172.21.15.130:0/2509936129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76441a2f90 con 0x7f76300035c0 2024-04-26T20:25:28.609 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.606+0000 7f763affd700 1 -- 172.21.15.130:0/2509936129 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76340044e0 con 0x7f76300035c0 2024-04-26T20:25:28.609 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.606+0000 7f763affd700 1 -- 172.21.15.130:0/2509936129 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7634011770 con 0x7f76300035c0 2024-04-26T20:25:28.609 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.607+0000 7f764c0c8700 1 -- 172.21.15.130:0/2509936129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7630002140 con 0x7f76300035c0 2024-04-26T20:25:28.609 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.607+0000 7f763affd700 1 -- 172.21.15.130:0/2509936129 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f7634023070 con 0x7f76300035c0 2024-04-26T20:25:28.609 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.607+0000 7f763affd700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f76280393f0 0x7f762803b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:28.609 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.607+0000 7f763affd700 1 -- 172.21.15.130:0/2509936129 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7634020050 con 0x7f76300035c0 2024-04-26T20:25:28.609 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.608+0000 7f7649663700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f76280393f0 0x7f762803b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:28.610 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.609+0000 7f7649663700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f76280393f0 0x7f762803b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7640009180 tx=0x7f7640008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:28.622 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.622+0000 7f763affd700 1 -- 172.21.15.130:0/2509936129 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7634028020 con 0x7f76300035c0 2024-04-26T20:25:28.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:28.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:28.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:28.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:28 smithi130 ceph-mon[32704]: Deploying daemon mon.b on smithi112 2024-04-26T20:25:28.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:28.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:28.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:28.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:28.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:28 smithi102 ceph-mon[28094]: Deploying daemon mon.b on smithi112 2024-04-26T20:25:28.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:28.904 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.903+0000 7f764c0c8700 1 -- 172.21.15.130:0/2509936129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7630002c30 con 0x7f76300035c0 2024-04-26T20:25:28.905 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.905+0000 7f763affd700 1 -- 172.21.15.130:0/2509936129 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f7634004c10 con 0x7f76300035c0 2024-04-26T20:25:28.905 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:25:28.906 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":2,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","modified":"2024-04-26T20:25:22.408227Z","created":"2024-04-26T20:23:15.907271Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T20:25:28.907 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 -- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f76280393f0 msgr2=0x7f762803b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:28.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f76280393f0 0x7f762803b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7640009180 tx=0x7f7640008040 comp rx=0 tx=0).stop 2024-04-26T20:25:28.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 -- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76300035c0 msgr2=0x7f7644199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:28.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76300035c0 0x7f7644199820 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7634010010 tx=0x7f76340116a0 comp rx=0 tx=0).stop 2024-04-26T20:25:28.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 -- 172.21.15.130:0/2509936129 shutdown_connections 2024-04-26T20:25:28.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f76280393f0 0x7f762803b8b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:28.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76300035c0 0x7f7644199820 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:28.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 --2- 172.21.15.130:0/2509936129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7644199d60 0x7f764419a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:28.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 -- 172.21.15.130:0/2509936129 >> 172.21.15.130:0/2509936129 conn(0x7f76440fd900 msgr2=0x7f76440febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:28.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 -- 172.21.15.130:0/2509936129 shutdown_connections 2024-04-26T20:25:28.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:28.907+0000 7f764c0c8700 1 -- 172.21.15.130:0/2509936129 wait complete. 2024-04-26T20:25:28.909 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 2 2024-04-26T20:25:29.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:29 smithi130 ceph-mon[32704]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:29.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:29 smithi130 ceph-mon[32704]: from='client.? 172.21.15.130:0/2509936129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T20:25:29.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:29 smithi102 ceph-mon[28094]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:29.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:29 smithi102 ceph-mon[28094]: from='client.? 172.21.15.130:0/2509936129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T20:25:30.561 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T20:25:30.561 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T20:25:30.802 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:25:36.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: mon.c calling monitor election 2024-04-26T20:25:36.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:25:36.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: mon.a calling monitor election 2024-04-26T20:25:36.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:36.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: monmap epoch 3 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: last_changed 2024-04-26T20:25:31.278658+0000 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: created 2024-04-26T20:23:15.907271+0000 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: min_mon_release 19 (squid) 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: election_strategy: 1 2024-04-26T20:25:36.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-26T20:25:36.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-26T20:25:36.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-04-26T20:25:36.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: fsmap 2024-04-26T20:25:36.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:25:36.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: mgrmap e13: x(active, since 67s) 2024-04-26T20:25:36.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: overall HEALTH_OK 2024-04-26T20:25:36.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:36.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:36.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: mon.c calling monitor election 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: mon.a calling monitor election 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T20:25:36.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: monmap epoch 3 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: last_changed 2024-04-26T20:25:31.278658+0000 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: created 2024-04-26T20:23:15.907271+0000 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: min_mon_release 19 (squid) 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: election_strategy: 1 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: fsmap 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: mgrmap e13: x(active, since 67s) 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: overall HEALTH_OK 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:36.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:36.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.658+0000 7fdb984d3700 1 -- 172.21.15.130:0/1129745127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c0057d0 msgr2=0x7fdb7c007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:36.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.658+0000 7fdb984d3700 1 --2- 172.21.15.130:0/1129745127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c0057d0 0x7fdb7c007c60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdb90101d40 tx=0x7fdb8c00fe60 comp rx=0 tx=0).stop 2024-04-26T20:25:36.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.658+0000 7fdb984d3700 1 -- 172.21.15.130:0/1129745127 shutdown_connections 2024-04-26T20:25:36.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.658+0000 7fdb984d3700 1 --2- 172.21.15.130:0/1129745127 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c0057d0 0x7fdb7c007c60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:36.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.658+0000 7fdb984d3700 1 -- 172.21.15.130:0/1129745127 >> 172.21.15.130:0/1129745127 conn(0x7fdb900fd940 msgr2=0x7fdb900ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:36.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.659+0000 7fdb984d3700 1 -- 172.21.15.130:0/1129745127 shutdown_connections 2024-04-26T20:25:36.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.659+0000 7fdb984d3700 1 -- 172.21.15.130:0/1129745127 wait complete. 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.659+0000 7fdb984d3700 1 Processor -- start 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.659+0000 7fdb984d3700 1 -- start start 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb984d3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c0057d0 0x7fdb901a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb984d3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb901a0f80 0x7fdb9019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb984d3700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb9019b050 0x7fdb9019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb984d3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90108cf0 con 0x7fdb7c0057d0 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb984d3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9006c4c0 con 0x7fdb901a0f80 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb984d3700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90108e70 con 0x7fdb9019b050 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb96a70700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb9019b050 0x7fdb9019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb96a70700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb9019b050 0x7fdb9019b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:47800/0 (socket says 172.21.15.130:47800) 2024-04-26T20:25:36.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb96a70700 1 -- 172.21.15.130:0/3214235708 learned_addr learned my addr 172.21.15.130:0/3214235708 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb96a70700 1 -- 172.21.15.130:0/3214235708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb9019b050 msgr2=0x7fdb9019b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb96a70700 1 -- 172.21.15.130:0/3214235708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb9019b050 msgr2=0x7fdb9019b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb96a70700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb9019b050 0x7fdb9019b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb96a70700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb9019b050 0x7fdb9019b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb95a6e700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb901a0f80 0x7fdb9019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb9626f700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c0057d0 0x7fdb901a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb95a6e700 1 -- 172.21.15.130:0/3214235708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb901a0f80 msgr2=0x7fdb9019ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb95a6e700 1 -- 172.21.15.130:0/3214235708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb901a0f80 msgr2=0x7fdb9019ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb95a6e700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb901a0f80 0x7fdb9019ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.660+0000 7fdb95a6e700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb901a0f80 0x7fdb9019ab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.661+0000 7fdb9626f700 1 -- 172.21.15.130:0/3214235708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb9019b050 msgr2=0x7fdb9019b500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.661+0000 7fdb9626f700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb9019b050 0x7fdb9019b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.661+0000 7fdb9626f700 1 -- 172.21.15.130:0/3214235708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb901a0f80 msgr2=0x7fdb9019ab10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.661+0000 7fdb9626f700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb901a0f80 0x7fdb9019ab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:36.663 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.661+0000 7fdb9626f700 1 -- 172.21.15.130:0/3214235708 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb901a77f0 con 0x7fdb7c0057d0 2024-04-26T20:25:36.664 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.661+0000 7fdb9626f700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c0057d0 0x7fdb901a0a40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdb90101d40 tx=0x7fdb8c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:36.665 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.661+0000 7fdb877fe700 1 -- 172.21.15.130:0/3214235708 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c02b020 con 0x7fdb7c0057d0 2024-04-26T20:25:36.665 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.662+0000 7fdb877fe700 1 -- 172.21.15.130:0/3214235708 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb8c0056c0 con 0x7fdb7c0057d0 2024-04-26T20:25:36.665 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.662+0000 7fdb877fe700 1 -- 172.21.15.130:0/3214235708 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c022040 con 0x7fdb7c0057d0 2024-04-26T20:25:36.665 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.662+0000 7fdb984d3700 1 -- 172.21.15.130:0/3214235708 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb901a7a20 con 0x7fdb7c0057d0 2024-04-26T20:25:36.665 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.662+0000 7fdb984d3700 1 -- 172.21.15.130:0/3214235708 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb901a7f60 con 0x7fdb7c0057d0 2024-04-26T20:25:36.665 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.663+0000 7fdb984d3700 1 -- 172.21.15.130:0/3214235708 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb90103550 con 0x7fdb7c0057d0 2024-04-26T20:25:36.665 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.663+0000 7fdb877fe700 1 -- 172.21.15.130:0/3214235708 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fdb8c02e020 con 0x7fdb7c0057d0 2024-04-26T20:25:36.665 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.663+0000 7fdb877fe700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdb74039ae0 0x7fdb7403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:36.665 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.663+0000 7fdb877fe700 1 -- 172.21.15.130:0/3214235708 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdb8c04f2a0 con 0x7fdb7c0057d0 2024-04-26T20:25:36.665 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.664+0000 7fdb95a6e700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdb74039ae0 0x7fdb7403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:36.666 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.665+0000 7fdb95a6e700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdb74039ae0 0x7fdb7403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdb80009850 tx=0x7fdb80008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:36.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.678+0000 7fdb877fe700 1 -- 172.21.15.130:0/3214235708 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdb8c035020 con 0x7fdb7c0057d0 2024-04-26T20:25:36.972 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.972+0000 7fdb984d3700 1 -- 172.21.15.130:0/3214235708 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fdb9002e490 con 0x7fdb7c0057d0 2024-04-26T20:25:36.974 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.973+0000 7fdb877fe700 1 -- 172.21.15.130:0/3214235708 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7fdb8c02f3b0 con 0x7fdb7c0057d0 2024-04-26T20:25:36.974 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-26T20:25:36.974 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":3,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","modified":"2024-04-26T20:25:31.278658Z","created":"2024-04-26T20:23:15.907271Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T20:25:36.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.975+0000 7fdb984d3700 1 -- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdb74039ae0 msgr2=0x7fdb7403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:36.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.975+0000 7fdb984d3700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdb74039ae0 0x7fdb7403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdb80009850 tx=0x7fdb80008040 comp rx=0 tx=0).stop 2024-04-26T20:25:36.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.975+0000 7fdb984d3700 1 -- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c0057d0 msgr2=0x7fdb901a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:36.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.975+0000 7fdb984d3700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c0057d0 0x7fdb901a0a40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdb90101d40 tx=0x7fdb8c003d00 comp rx=0 tx=0).stop 2024-04-26T20:25:36.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.975+0000 7fdb984d3700 1 -- 172.21.15.130:0/3214235708 shutdown_connections 2024-04-26T20:25:36.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.975+0000 7fdb984d3700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdb74039ae0 0x7fdb7403bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:36.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.975+0000 7fdb984d3700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb9019b050 0x7fdb9019b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:36.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.975+0000 7fdb984d3700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c0057d0 0x7fdb901a0a40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:36.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.975+0000 7fdb984d3700 1 --2- 172.21.15.130:0/3214235708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb901a0f80 0x7fdb9019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:36.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.975+0000 7fdb984d3700 1 -- 172.21.15.130:0/3214235708 >> 172.21.15.130:0/3214235708 conn(0x7fdb900fd940 msgr2=0x7fdb900fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:36.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.976+0000 7fdb984d3700 1 -- 172.21.15.130:0/3214235708 shutdown_connections 2024-04-26T20:25:36.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:36.976+0000 7fdb984d3700 1 -- 172.21.15.130:0/3214235708 wait complete. 2024-04-26T20:25:36.977 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 3 2024-04-26T20:25:37.469 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T20:25:37.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T20:25:37.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:37 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:37.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:37 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:37.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:37 smithi130 ceph-mon[32704]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:37.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:37 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:37.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:37 smithi130 ceph-mon[32704]: from='client.? 172.21.15.130:0/3214235708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T20:25:37.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:37 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:37.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:37 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:37.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:37 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:37.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:37 smithi102 ceph-mon[28094]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:37.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:37 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:37.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:37 smithi102 ceph-mon[28094]: from='client.? 172.21.15.130:0/3214235708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T20:25:37.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:37 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:37.914 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:25:38.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:39.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: mon.b calling monitor election 2024-04-26T20:25:39.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: mon.b calling monitor election 2024-04-26T20:25:39.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: mon.c calling monitor election 2024-04-26T20:25:39.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: mon.a calling monitor election 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: monmap epoch 3 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: last_changed 2024-04-26T20:25:31.278658+0000 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: created 2024-04-26T20:23:15.907271+0000 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: min_mon_release 19 (squid) 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: election_strategy: 1 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: fsmap 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: mgrmap e13: x(active, since 69s) 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: overall HEALTH_OK 2024-04-26T20:25:39.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:39.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:39.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:39.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:39.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-26T20:25:39.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-26T20:25:39.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-26T20:25:39.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:39.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:39.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: mon.b calling monitor election 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: mon.b calling monitor election 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: mon.c calling monitor election 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: mon.a calling monitor election 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: monmap epoch 3 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: last_changed 2024-04-26T20:25:31.278658+0000 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: created 2024-04-26T20:23:15.907271+0000 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: min_mon_release 19 (squid) 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: election_strategy: 1 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-26T20:25:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: fsmap 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: mgrmap e13: x(active, since 69s) 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: overall HEALTH_OK 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:39.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: mon.b calling monitor election 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: mon.b calling monitor election 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: mon.c calling monitor election 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: mon.a calling monitor election 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: monmap epoch 3 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: last_changed 2024-04-26T20:25:31.278658+0000 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: created 2024-04-26T20:23:15.907271+0000 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: min_mon_release 19 (squid) 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: election_strategy: 1 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-26T20:25:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: fsmap 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: mgrmap e13: x(active, since 69s) 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: overall HEALTH_OK 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-26T20:25:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:39.374 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.003+0000 7ff22aa00700 1 -- 172.21.15.102:0/2705480873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff224103060 msgr2=0x7ff224103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.003+0000 7ff22aa00700 1 --2- 172.21.15.102:0/2705480873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff224103060 0x7ff224103440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff214009910 tx=0x7ff214009c20 comp rx=0 tx=0).stop 2024-04-26T20:25:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.004+0000 7ff22aa00700 1 -- 172.21.15.102:0/2705480873 shutdown_connections 2024-04-26T20:25:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.004+0000 7ff22aa00700 1 --2- 172.21.15.102:0/2705480873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff224103060 0x7ff224103440 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.004+0000 7ff22aa00700 1 -- 172.21.15.102:0/2705480873 >> 172.21.15.102:0/2705480873 conn(0x7ff2240fe910 msgr2=0x7ff224100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.004+0000 7ff22aa00700 1 -- 172.21.15.102:0/2705480873 shutdown_connections 2024-04-26T20:25:40.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.004+0000 7ff22aa00700 1 -- 172.21.15.102:0/2705480873 wait complete. 2024-04-26T20:25:40.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.005+0000 7ff22aa00700 1 Processor -- start 2024-04-26T20:25:40.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.005+0000 7ff22aa00700 1 -- start start 2024-04-26T20:25:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff22aa00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff224103060 0x7ff2241a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff22aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2241a0f80 0x7ff22419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff22aa00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff22419b050 0x7ff22419b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff22aa00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff22410d2e0 con 0x7ff2241a0f80 2024-04-26T20:25:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff22aa00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff22410d160 con 0x7ff224103060 2024-04-26T20:25:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff22aa00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff22410d460 con 0x7ff22419b050 2024-04-26T20:25:40.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff21be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2241a0f80 0x7ff22419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:40.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff223e00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff224103060 0x7ff2241a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:40.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff21be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2241a0f80 0x7ff22419ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44968/0 (socket says 172.21.15.102:44968) 2024-04-26T20:25:40.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff21be00700 1 -- 172.21.15.102:0/1006845155 learned_addr learned my addr 172.21.15.102:0/1006845155 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:25:40.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff228a00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff22419b050 0x7ff22419b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.006+0000 7ff21be00700 1 -- 172.21.15.102:0/1006845155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff22419b050 msgr2=0x7ff22419b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.007+0000 7ff21be00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff22419b050 0x7ff22419b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.007+0000 7ff21be00700 1 -- 172.21.15.102:0/1006845155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff224103060 msgr2=0x7ff2241a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.007+0000 7ff21be00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff224103060 0x7ff2241a0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.007+0000 7ff21be00700 1 -- 172.21.15.102:0/1006845155 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2241a77f0 con 0x7ff2241a0f80 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.007+0000 7ff21be00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2241a0f80 0x7ff22419ab10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00b780 tx=0x7ff20c00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.007+0000 7ff221600700 1 -- 172.21.15.102:0/1006845155 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c016070 con 0x7ff2241a0f80 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.007+0000 7ff221600700 1 -- 172.21.15.102:0/1006845155 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff20c0036a0 con 0x7ff2241a0f80 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.007+0000 7ff221600700 1 -- 172.21.15.102:0/1006845155 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0119c0 con 0x7ff2241a0f80 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.007+0000 7ff22aa00700 1 -- 172.21.15.102:0/1006845155 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2241a7a80 con 0x7ff2241a0f80 2024-04-26T20:25:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.007+0000 7ff22aa00700 1 -- 172.21.15.102:0/1006845155 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2241a8010 con 0x7ff2241a0f80 2024-04-26T20:25:40.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.009+0000 7ff221600700 1 -- 172.21.15.102:0/1006845155 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7ff20c019070 con 0x7ff2241a0f80 2024-04-26T20:25:40.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.009+0000 7ff22aa00700 1 -- 172.21.15.102:0/1006845155 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff224104540 con 0x7ff2241a0f80 2024-04-26T20:25:40.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.009+0000 7ff221600700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff208039ae0 0x7ff20803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:40.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.009+0000 7ff221600700 1 -- 172.21.15.102:0/1006845155 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff20c04b4b0 con 0x7ff2241a0f80 2024-04-26T20:25:40.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.010+0000 7ff223e00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff208039ae0 0x7ff20803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:40.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.010+0000 7ff223e00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff208039ae0 0x7ff20803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff214006010 tx=0x7ff21401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:40.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.015+0000 7ff221600700 1 -- 172.21.15.102:0/1006845155 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff20c0038c0 con 0x7ff2241a0f80 2024-04-26T20:25:40.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: Updating smithi112:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: Updating smithi130:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: Updating smithi102:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:40.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:25:40.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.247+0000 7ff22aa00700 1 -- 172.21.15.102:0/1006845155 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ff2241a82e0 con 0x7ff2241a0f80 2024-04-26T20:25:40.249 INFO:teuthology.orchestra.run.smithi102.stdout:# minimal ceph.conf for a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:25:40.249 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-04-26T20:25:40.249 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:25:40.249 INFO:teuthology.orchestra.run.smithi102.stdout: mon_host = [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] 2024-04-26T20:25:40.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.247+0000 7ff221600700 1 -- 172.21.15.102:0/1006845155 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7ff20c01a8d0 con 0x7ff2241a0f80 2024-04-26T20:25:40.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.250+0000 7ff22aa00700 1 -- 172.21.15.102:0/1006845155 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff208039ae0 msgr2=0x7ff20803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:40.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.250+0000 7ff22aa00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff208039ae0 0x7ff20803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff214006010 tx=0x7ff21401a040 comp rx=0 tx=0).stop 2024-04-26T20:25:40.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.250+0000 7ff22aa00700 1 -- 172.21.15.102:0/1006845155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2241a0f80 msgr2=0x7ff22419ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:40.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.250+0000 7ff22aa00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2241a0f80 0x7ff22419ab10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00b780 tx=0x7ff20c00ba90 comp rx=0 tx=0).stop 2024-04-26T20:25:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.251+0000 7ff22aa00700 1 -- 172.21.15.102:0/1006845155 shutdown_connections 2024-04-26T20:25:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.251+0000 7ff22aa00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff208039ae0 0x7ff20803bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.251+0000 7ff22aa00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff22419b050 0x7ff22419b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.251+0000 7ff22aa00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff224103060 0x7ff2241a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.251+0000 7ff22aa00700 1 --2- 172.21.15.102:0/1006845155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2241a0f80 0x7ff22419ab10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.251+0000 7ff22aa00700 1 -- 172.21.15.102:0/1006845155 >> 172.21.15.102:0/1006845155 conn(0x7ff2240fe910 msgr2=0x7ff224075790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.251+0000 7ff22aa00700 1 -- 172.21.15.102:0/1006845155 shutdown_connections 2024-04-26T20:25:40.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:40.251+0000 7ff22aa00700 1 -- 172.21.15.102:0/1006845155 wait complete. 2024-04-26T20:25:40.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: Updating smithi112:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:40.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: Updating smithi130:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: Updating smithi102:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:40.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:25:40.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: Updating smithi112:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:40.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: Updating smithi130:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:40.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: Updating smithi102:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T20:25:40.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:25:40.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:40.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:40.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:25:41.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:40 smithi102 ceph-mon[28094]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T20:25:41.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:41.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:40 smithi102 ceph-mon[28094]: Reconfiguring daemon mon.a on smithi102 2024-04-26T20:25:41.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:40 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1006845155' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:41.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:40 smithi102 ceph-mon[28094]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:41.093 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T20:25:41.094 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:25:41.094 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T20:25:41.174 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:25:41.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:25:41.272 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:40 smithi130 ceph-mon[32704]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T20:25:41.272 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:41.273 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:40 smithi130 ceph-mon[32704]: Reconfiguring daemon mon.a on smithi102 2024-04-26T20:25:41.273 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:40 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1006845155' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:41.273 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:40 smithi130 ceph-mon[32704]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:41.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:40 smithi112 ceph-mon[32769]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T20:25:41.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:41.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:40 smithi112 ceph-mon[32769]: Reconfiguring daemon mon.a on smithi102 2024-04-26T20:25:41.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:40 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1006845155' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:41.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:40 smithi112 ceph-mon[32769]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:41.290 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:25:41.290 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T20:25:41.322 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:25:41.322 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:25:41.389 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:25:41.389 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T20:25:41.421 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:25:41.421 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T20:25:41.487 INFO:tasks.cephadm:Adding mgr.x on smithi102 2024-04-26T20:25:41.487 INFO:tasks.cephadm:Adding mgr.y on smithi112 2024-04-26T20:25:41.487 INFO:tasks.cephadm:Adding mgr.z on smithi130 2024-04-26T20:25:41.488 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi102=x;smithi112=y;smithi130=z' 2024-04-26T20:25:41.761 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:25:43.100 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.099+0000 7f653796d700 1 -- 172.21.15.130:0/1053390413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6530103ac0 msgr2=0x7f6530105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:43.101 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.099+0000 7f653796d700 1 --2- 172.21.15.130:0/1053390413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6530103ac0 0x7f6530105eb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f652c00b0a0 tx=0x7f652c00b3b0 comp rx=0 tx=0).stop 2024-04-26T20:25:43.101 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.099+0000 7f653796d700 1 -- 172.21.15.130:0/1053390413 shutdown_connections 2024-04-26T20:25:43.101 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.100+0000 7f653796d700 1 --2- 172.21.15.130:0/1053390413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6530103ac0 0x7f6530105eb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:43.101 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.100+0000 7f653796d700 1 -- 172.21.15.130:0/1053390413 >> 172.21.15.130:0/1053390413 conn(0x7f65300fd430 msgr2=0x7f65300ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:43.101 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.100+0000 7f653796d700 1 -- 172.21.15.130:0/1053390413 shutdown_connections 2024-04-26T20:25:43.101 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.100+0000 7f653796d700 1 -- 172.21.15.130:0/1053390413 wait complete. 2024-04-26T20:25:43.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.100+0000 7f653796d700 1 Processor -- start 2024-04-26T20:25:43.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.100+0000 7f653796d700 1 -- start start 2024-04-26T20:25:43.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f653796d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6530103ac0 0x7f65301986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:43.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f653796d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6530198c00 0x7f653019dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:43.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f653796d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6530199080 0x7f65301a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:43.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f653796d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6530108e80 con 0x7f6530103ac0 2024-04-26T20:25:43.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f653796d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6530108d00 con 0x7f6530198c00 2024-04-26T20:25:43.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f653796d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6530109000 con 0x7f6530199080 2024-04-26T20:25:43.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6534f08700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6530198c00 0x7f653019dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6534f08700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6530198c00 0x7f653019dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.130:50200/0 (socket says 172.21.15.130:50200) 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6534f08700 1 -- 172.21.15.130:0/167384078 learned_addr learned my addr 172.21.15.130:0/167384078 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6535709700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6530103ac0 0x7f65301986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6535f0a700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6530199080 0x7f65301a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6534f08700 1 -- 172.21.15.130:0/167384078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6530199080 msgr2=0x7f65301a0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6534f08700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6530199080 0x7f65301a0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6534f08700 1 -- 172.21.15.130:0/167384078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6530103ac0 msgr2=0x7f65301986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6534f08700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6530103ac0 0x7f65301986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6534f08700 1 -- 172.21.15.130:0/167384078 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65301a0ad0 con 0x7f6530198c00 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.101+0000 7f6534f08700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6530198c00 0x7f653019dc70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f652000b780 tx=0x7f652000ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.102+0000 7f65267fc700 1 -- 172.21.15.130:0/167384078 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520004230 con 0x7f6530198c00 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.102+0000 7f65267fc700 1 -- 172.21.15.130:0/167384078 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65200043d0 con 0x7f6530198c00 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.102+0000 7f65267fc700 1 -- 172.21.15.130:0/167384078 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520011b20 con 0x7f6530198c00 2024-04-26T20:25:43.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.102+0000 7f653796d700 1 -- 172.21.15.130:0/167384078 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65301a0dc0 con 0x7f6530198c00 2024-04-26T20:25:43.106 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.102+0000 7f653796d700 1 -- 172.21.15.130:0/167384078 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65301a12f0 con 0x7f6530198c00 2024-04-26T20:25:43.106 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.103+0000 7f653796d700 1 -- 172.21.15.130:0/167384078 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6530064a80 con 0x7f6530198c00 2024-04-26T20:25:43.106 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.103+0000 7f65267fc700 1 -- 172.21.15.130:0/167384078 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f65200036a0 con 0x7f6530198c00 2024-04-26T20:25:43.106 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.103+0000 7f65267fc700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f651c039ae0 0x7f651c03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:43.106 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.103+0000 7f65267fc700 1 -- 172.21.15.130:0/167384078 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f652004c3f0 con 0x7f6530198c00 2024-04-26T20:25:43.106 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.103+0000 7f6535709700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f651c039ae0 0x7f651c03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:43.106 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.104+0000 7f6535709700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f651c039ae0 0x7f651c03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f652c007600 tx=0x7f652c0097c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:43.109 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.108+0000 7f65267fc700 1 -- 172.21.15.130:0/167384078 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6520020910 con 0x7f6530198c00 2024-04-26T20:25:43.321 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.320+0000 7f653796d700 1 -- 172.21.15.130:0/167384078 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi102=x;smithi112=y;smithi130=z", "target": ["mon-mgr", ""]}) -- 0x7f6530063860 con 0x7f651c039ae0 2024-04-26T20:25:43.335 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.335+0000 7f65267fc700 1 -- 172.21.15.130:0/167384078 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6530063860 con 0x7f651c039ae0 2024-04-26T20:25:43.336 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mgr update... 2024-04-26T20:25:43.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 -- 172.21.15.130:0/167384078 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f651c039ae0 msgr2=0x7f651c03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:43.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f651c039ae0 0x7f651c03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f652c007600 tx=0x7f652c0097c0 comp rx=0 tx=0).stop 2024-04-26T20:25:43.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 -- 172.21.15.130:0/167384078 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6530198c00 msgr2=0x7f653019dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:43.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6530198c00 0x7f653019dc70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f652000b780 tx=0x7f652000ba90 comp rx=0 tx=0).stop 2024-04-26T20:25:43.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 -- 172.21.15.130:0/167384078 shutdown_connections 2024-04-26T20:25:43.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f651c039ae0 0x7f651c03bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:43.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6530199080 0x7f65301a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:43.338 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6530103ac0 0x7f65301986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:43.339 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 --2- 172.21.15.130:0/167384078 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6530198c00 0x7f653019dc70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:43.339 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 -- 172.21.15.130:0/167384078 >> 172.21.15.130:0/167384078 conn(0x7f65300fd430 msgr2=0x7f65300fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:43.339 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 -- 172.21.15.130:0/167384078 shutdown_connections 2024-04-26T20:25:43.339 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:25:43.337+0000 7f653796d700 1 -- 172.21.15.130:0/167384078 wait complete. 2024-04-26T20:25:43.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:43.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:43.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:43 smithi112 ceph-mon[32769]: Reconfiguring mon.b (monmap changed)... 2024-04-26T20:25:43.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:43.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:25:43.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:43.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:43 smithi112 ceph-mon[32769]: Reconfiguring daemon mon.b on smithi112 2024-04-26T20:25:43.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:43 smithi112 ceph-mon[32769]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:43.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:43.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:43.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:43 smithi130 ceph-mon[32704]: Reconfiguring mon.b (monmap changed)... 2024-04-26T20:25:43.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:43.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:25:43.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:43.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:43 smithi130 ceph-mon[32704]: Reconfiguring daemon mon.b on smithi112 2024-04-26T20:25:43.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:43 smithi130 ceph-mon[32704]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:43.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:43.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:43.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:43 smithi102 ceph-mon[28094]: Reconfiguring mon.b (monmap changed)... 2024-04-26T20:25:43.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:43.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:25:43.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:43.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:43 smithi102 ceph-mon[28094]: Reconfiguring daemon mon.b on smithi112 2024-04-26T20:25:43.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:43 smithi102 ceph-mon[28094]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:43.945 DEBUG:teuthology.orchestra.run.smithi112:mgr.y> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T20:25:43.947 DEBUG:teuthology.orchestra.run.smithi130:mgr.z> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T20:25:43.949 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T20:25:43.950 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:25:43.950 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T20:25:44.010 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T20:25:44.010 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-04-26T20:25:44.063 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T20:25:44.063 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:44.063 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 634 Links: 1 2024-04-26T20:25:44.063 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:44.064 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:44.064 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-26 20:24:45.320222356 +0000 2024-04-26T20:25:44.064 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-26 20:18:50.106312906 +0000 2024-04-26T20:25:44.064 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-26 20:18:50.106312906 +0000 2024-04-26T20:25:44.064 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-26 20:18:50.106312906 +0000 2024-04-26T20:25:44.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T20:25:44.146 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-26T20:25:44.147 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-26T20:25:44.147 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000356697 s, 1.4 MB/s 2024-04-26T20:25:44.151 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T20:25:44.203 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-04-26T20:25:44.253 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T20:25:44.253 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:44.253 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-26T20:25:44.253 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:44.253 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:44.253 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-26 20:24:45.376221226 +0000 2024-04-26T20:25:44.253 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-26 20:18:50.695300996 +0000 2024-04-26T20:25:44.253 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-26 20:18:50.695300996 +0000 2024-04-26T20:25:44.254 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-26 20:18:50.695300996 +0000 2024-04-26T20:25:44.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T20:25:44.330 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-26T20:25:44.330 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-26T20:25:44.331 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000395494 s, 1.3 MB/s 2024-04-26T20:25:44.334 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T20:25:44.386 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-04-26T20:25:44.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:44 smithi102 ceph-mon[28094]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi102=x;smithi112=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:44.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:44 smithi102 ceph-mon[28094]: Saving service mgr spec with placement smithi102=x;smithi112=y;smithi130=z;count:3 2024-04-26T20:25:44.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:44.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:44.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:44.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:44.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:25:44.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:44.435 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T20:25:44.435 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:44.436 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-04-26T20:25:44.436 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:44.436 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:44.436 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-26 20:24:45.431220117 +0000 2024-04-26T20:25:44.436 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-26 20:18:51.292288925 +0000 2024-04-26T20:25:44.436 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-26 20:18:51.292288925 +0000 2024-04-26T20:25:44.436 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-26 20:18:51.292288925 +0000 2024-04-26T20:25:44.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T20:25:44.562 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-26T20:25:44.562 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-26T20:25:44.562 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000393435 s, 1.3 MB/s 2024-04-26T20:25:44.566 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T20:25:44.594 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:44 smithi130 ceph-mon[32704]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi102=x;smithi112=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:44.595 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:44 smithi130 ceph-mon[32704]: Saving service mgr spec with placement smithi102=x;smithi112=y;smithi130=z;count:3 2024-04-26T20:25:44.595 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:44.595 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:44.595 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:44.595 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:44.595 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:25:44.595 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:44.618 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-04-26T20:25:44.668 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T20:25:44.668 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:44.668 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 691 Links: 1 2024-04-26T20:25:44.668 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:44.668 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:44.668 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-26 20:24:45.482219088 +0000 2024-04-26T20:25:44.668 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-26 20:18:51.897276692 +0000 2024-04-26T20:25:44.668 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-26 20:18:51.897276692 +0000 2024-04-26T20:25:44.668 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-26 20:18:51.897276692 +0000 2024-04-26T20:25:44.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T20:25:44.745 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-26T20:25:44.746 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-26T20:25:44.746 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000344574 s, 1.5 MB/s 2024-04-26T20:25:44.749 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T20:25:44.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:44 smithi112 ceph-mon[32769]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi102=x;smithi112=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:44.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:44 smithi112 ceph-mon[32769]: Saving service mgr spec with placement smithi102=x;smithi112=y;smithi130=z;count:3 2024-04-26T20:25:44.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:44.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:44.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:44.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T20:25:44.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T20:25:44.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:44.805 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-26T20:25:44.805 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T20:25:44.824 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T20:25:44.824 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-04-26T20:25:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T20:25:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:44.881 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-26T20:25:44.881 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:44.881 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:44.881 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-04-26 20:25:30.735073320 +0000 2024-04-26T20:25:44.881 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-04-26 20:17:47.569135194 +0000 2024-04-26T20:25:44.882 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-04-26 20:17:47.569135194 +0000 2024-04-26T20:25:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-04-26 20:17:47.569135194 +0000 2024-04-26T20:25:44.882 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T20:25:44.946 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-04-26T20:25:44.947 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-04-26T20:25:44.947 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000224641 s, 2.3 MB/s 2024-04-26T20:25:44.948 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T20:25:45.005 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-04-26T20:25:45.061 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T20:25:45.061 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:45.061 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-26T20:25:45.061 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:45.061 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:45.061 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-04-26 20:25:30.735073320 +0000 2024-04-26T20:25:45.061 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-04-26 20:17:47.914125452 +0000 2024-04-26T20:25:45.061 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-04-26 20:17:47.914125452 +0000 2024-04-26T20:25:45.061 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-04-26 20:17:47.914125452 +0000 2024-04-26T20:25:45.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T20:25:45.125 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-04-26T20:25:45.125 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-04-26T20:25:45.125 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000237066 s, 2.2 MB/s 2024-04-26T20:25:45.127 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T20:25:45.184 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-04-26T20:25:45.240 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T20:25:45.240 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:45.240 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-26T20:25:45.240 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:45.240 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:45.240 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-04-26 20:25:30.736073293 +0000 2024-04-26T20:25:45.240 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-04-26 20:17:48.257115767 +0000 2024-04-26T20:25:45.241 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-04-26 20:17:48.257115767 +0000 2024-04-26T20:25:45.241 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-04-26 20:17:48.257115767 +0000 2024-04-26T20:25:45.241 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T20:25:45.303 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-04-26T20:25:45.303 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-04-26T20:25:45.303 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000255788 s, 2.0 MB/s 2024-04-26T20:25:45.304 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T20:25:45.361 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-04-26T20:25:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T20:25:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:45.416 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-26T20:25:45.416 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:45.416 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:45.416 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-04-26 20:25:30.736073293 +0000 2024-04-26T20:25:45.416 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-04-26 20:17:48.578106703 +0000 2024-04-26T20:25:45.416 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-04-26 20:17:48.578106703 +0000 2024-04-26T20:25:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-04-26 20:17:48.578106703 +0000 2024-04-26T20:25:45.417 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T20:25:45.478 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-04-26T20:25:45.478 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-04-26T20:25:45.478 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000175184 s, 2.9 MB/s 2024-04-26T20:25:45.479 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T20:25:45.534 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-26T20:25:45.534 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T20:25:45.549 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T20:25:45.549 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_1 2024-04-26T20:25:45.604 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T20:25:45.604 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:45.604 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-26T20:25:45.604 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:45.604 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:45.604 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-26 20:25:21.562754000 +0000 2024-04-26T20:25:45.605 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-26 20:17:47.862263084 +0000 2024-04-26T20:25:45.605 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-26 20:17:47.862263084 +0000 2024-04-26T20:25:45.605 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-04-26 20:17:47.862263084 +0000 2024-04-26T20:25:45.605 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T20:25:45.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: Reconfiguring mon.c (monmap changed)... 2024-04-26T20:25:45.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: Reconfiguring daemon mon.c on smithi130 2024-04-26T20:25:45.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:45.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:45.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:45.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:45.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:45.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:45.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:45.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:25:45.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:25:45.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:25:45.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:45.645 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-26T20:25:45.645 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-26T20:25:45.645 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000612061 s, 837 kB/s 2024-04-26T20:25:45.646 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: Reconfiguring mon.c (monmap changed)... 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: Reconfiguring daemon mon.c on smithi130 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:25:45.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:45.702 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_2 2024-04-26T20:25:45.757 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T20:25:45.757 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:45.757 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-26T20:25:45.757 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:45.757 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:45.757 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-26 20:25:21.562754000 +0000 2024-04-26T20:25:45.757 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-26 20:17:48.198256624 +0000 2024-04-26T20:25:45.757 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-26 20:17:48.198256624 +0000 2024-04-26T20:25:45.757 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-04-26 20:17:48.198256624 +0000 2024-04-26T20:25:45.758 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T20:25:45.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: Reconfiguring mon.c (monmap changed)... 2024-04-26T20:25:45.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: Reconfiguring daemon mon.c on smithi130 2024-04-26T20:25:45.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:45.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:45.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:45.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:45.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:45.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:45.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:45.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:25:45.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:25:45.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:25:45.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:45.820 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-26T20:25:45.820 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-26T20:25:45.820 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.00021104 s, 2.4 MB/s 2024-04-26T20:25:45.822 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T20:25:45.878 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_3 2024-04-26T20:25:45.932 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T20:25:45.933 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:45.933 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-26T20:25:45.933 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:45.933 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:45.933 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-26 20:25:21.563753982 +0000 2024-04-26T20:25:45.933 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-26 20:17:48.548249895 +0000 2024-04-26T20:25:45.933 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-26 20:17:48.548249895 +0000 2024-04-26T20:25:45.933 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-04-26 20:17:48.548249895 +0000 2024-04-26T20:25:45.933 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T20:25:45.995 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-26T20:25:45.995 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-26T20:25:45.995 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000254269 s, 2.0 MB/s 2024-04-26T20:25:45.996 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T20:25:46.052 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_4 2024-04-26T20:25:46.107 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T20:25:46.107 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T20:25:46.107 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-26T20:25:46.107 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T20:25:46.107 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T20:25:46.107 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-26 20:25:21.564753963 +0000 2024-04-26T20:25:46.107 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-26 20:17:48.885243416 +0000 2024-04-26T20:25:46.107 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-26 20:17:48.885243416 +0000 2024-04-26T20:25:46.108 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-04-26 20:17:48.885243416 +0000 2024-04-26T20:25:46.108 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T20:25:46.171 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-26T20:25:46.171 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-26T20:25:46.172 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000234869 s, 2.2 MB/s 2024-04-26T20:25:46.173 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T20:25:46.228 INFO:tasks.cephadm:Deploying osd.0 on smithi102 with /dev/vg_nvme/lv_4... 2024-04-26T20:25:46.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T20:25:46.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:46 smithi130 ceph-mon[32704]: Deploying daemon mgr.z on smithi130 2024-04-26T20:25:46.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:46 smithi102 ceph-mon[28094]: Deploying daemon mgr.z on smithi130 2024-04-26T20:25:46.667 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:25:46.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:46 smithi112 ceph-mon[32769]: Deploying daemon mgr.z on smithi130 2024-04-26T20:25:47.520 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:47 smithi130 ceph-mon[32704]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:47.580 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:47 smithi102 ceph-mon[28094]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:47.678 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:47 smithi112 ceph-mon[32769]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:47.792 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:47 smithi130 podman[34565]: 2024-04-26 20:25:47.768067914 +0000 UTC m=+0.363563468 container start 92d42ddf2dc52f60fbba0ceba9ddde80f2c0e7070e5fdafbd583a53d81a9c884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:25:48.104 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:47 smithi130 ceph-mgr[34584]: -- 172.21.15.130:0/3190104630 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564bfe05c5a0 con 0x564bfe030800 2024-04-26T20:25:48.104 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:47 smithi130 bash[34565]: 92d42ddf2dc52f60fbba0ceba9ddde80f2c0e7070e5fdafbd583a53d81a9c884 2024-04-26T20:25:48.104 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:47 smithi130 systemd[1]: Started Ceph mgr.z for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:25:48.104 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:47 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:47.920+0000 7f93df541200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T20:25:48.104 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:48.103+0000 7f93df541200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:25:48.371 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:48.203+0000 7f93df541200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T20:25:48.871 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:48.506+0000 7f93df541200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T20:25:48.871 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:48.589+0000 7f93df541200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T20:25:49.227 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:48 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:49.227 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:48 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:49.227 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:48 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:49.228 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:48 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:25:49.228 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:48 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:25:49.228 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:48 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:25:49.228 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:48 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:49.228 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:48 smithi112 ceph-mon[32769]: Deploying daemon mgr.y on smithi112 2024-04-26T20:25:49.228 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:48 smithi112 ceph-mon[32769]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:49.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:49.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:49.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:49.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:25:49.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:25:49.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:25:49.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:49.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-mon[32704]: Deploying daemon mgr.y on smithi112 2024-04-26T20:25:49.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:48 smithi130 ceph-mon[32704]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:49.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:48 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:49.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:48 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:49.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:48 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:49.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:48 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:25:49.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:48 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T20:25:49.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:48 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:25:49.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:48 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:49.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:48 smithi102 ceph-mon[28094]: Deploying daemon mgr.y on smithi112 2024-04-26T20:25:49.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:48 smithi102 ceph-mon[28094]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:49.448 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T20:25:49.449 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T20:25:49.449 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-04-26T20:25:49.449 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-04-26T20:25:49.449 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0353942 s, 296 MB/s 2024-04-26T20:25:49.449 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-04-26T20:25:50.027 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:49 smithi112 systemd[1]: Starting Ceph mgr.y for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:25:50.104 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:49.767+0000 7f93df541200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T20:25:50.104 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:49.933+0000 7f93df541200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T20:25:50.104 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:50.103+0000 7f93df541200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T20:25:50.206 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_4 2024-04-26T20:25:50.397 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:50 smithi112 podman[34158]: 2024-04-26 20:25:50.143602327 +0000 UTC m=+0.112286325 container create 95b0502d8a59a26e0844e908eb8c64f9b14dbcb55638761bd2d6efe9403cb345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-26T20:25:50.398 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:50 smithi112 podman[34158]: 2024-04-26 20:25:50.049149668 +0000 UTC m=+0.017833676 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:25:50.398 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:50 smithi112 podman[34158]: 2024-04-26 20:25:50.397829474 +0000 UTC m=+0.366513475 container init 95b0502d8a59a26e0844e908eb8c64f9b14dbcb55638761bd2d6efe9403cb345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T20:25:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:25:50.678 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:50.384+0000 7f93df541200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T20:25:50.679 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:50.475+0000 7f93df541200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T20:25:50.679 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:50.678+0000 7f93df541200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T20:25:50.756 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:50 smithi112 podman[34158]: 2024-04-26 20:25:50.400475744 +0000 UTC m=+0.369159743 container start 95b0502d8a59a26e0844e908eb8c64f9b14dbcb55638761bd2d6efe9403cb345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:25:50.756 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:50 smithi112 ceph-mgr[34177]: -- 172.21.15.112:0/2300900322 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f7d24905a0 con 0x55f7d2464800 2024-04-26T20:25:50.756 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:50 smithi112 bash[34158]: 95b0502d8a59a26e0844e908eb8c64f9b14dbcb55638761bd2d6efe9403cb345 2024-04-26T20:25:50.757 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:50 smithi112 systemd[1]: Started Ceph mgr.y for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:25:50.757 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:50 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:50.566+0000 7f0c44246200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T20:25:50.757 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:50 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:50.756+0000 7f0c44246200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:25:51.026 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:50 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:50.858+0000 7f0c44246200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T20:25:51.121 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:50.871+0000 7f93df541200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:25:51.526 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:51 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:51.183+0000 7f0c44246200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T20:25:51.527 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:51 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:51.270+0000 7f0c44246200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T20:25:51.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:51 smithi102 ceph-mon[28094]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:51.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.787 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:51.859 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:51 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:51.509+0000 7f93df541200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T20:25:51.859 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:51 smithi130 ceph-mon[32704]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:51.859 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.859 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.860 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.860 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.860 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:51.861 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:51 smithi112 ceph-mon[32769]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:51.862 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.862 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.862 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.862 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:51.862 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:52.121 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:51 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:51.935+0000 7f93df541200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T20:25:52.121 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:52 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:52.031+0000 7f93df541200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T20:25:52.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.552+0000 7f9a39600700 1 -- 172.21.15.102:0/711949035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a34077450 msgr2=0x7f9a340758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:52.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.552+0000 7f9a39600700 1 --2- 172.21.15.102:0/711949035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a34077450 0x7f9a340758b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9a240098f0 tx=0x7f9a24009c00 comp rx=0 tx=0).stop 2024-04-26T20:25:52.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.552+0000 7f9a39600700 1 -- 172.21.15.102:0/711949035 shutdown_connections 2024-04-26T20:25:52.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.552+0000 7f9a39600700 1 --2- 172.21.15.102:0/711949035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a3410b360 0x7f9a3410d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:52.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.552+0000 7f9a39600700 1 --2- 172.21.15.102:0/711949035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a34077450 0x7f9a340758b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:52.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.552+0000 7f9a39600700 1 --2- 172.21.15.102:0/711949035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a34075e80 0x7f9a34076300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:52.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.552+0000 7f9a39600700 1 -- 172.21.15.102:0/711949035 >> 172.21.15.102:0/711949035 conn(0x7f9a340fe9f0 msgr2=0x7f9a34100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:25:52.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.552+0000 7f9a39600700 1 -- 172.21.15.102:0/711949035 shutdown_connections 2024-04-26T20:25:52.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.552+0000 7f9a39600700 1 -- 172.21.15.102:0/711949035 wait complete. 2024-04-26T20:25:52.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.553+0000 7f9a39600700 1 Processor -- start 2024-04-26T20:25:52.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.553+0000 7f9a39600700 1 -- start start 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a39600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a34075e80 0x7f9a3419b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a39600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a34077450 0x7f9a3419bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a39600700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a3410b360 0x7f9a341a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a39600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34111110 con 0x7f9a34075e80 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a39600700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34110f90 con 0x7f9a3410b360 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a39600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34111290 con 0x7f9a34077450 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a32a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a34075e80 0x7f9a3419b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a32000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a34077450 0x7f9a3419bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a32000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a34077450 0x7f9a3419bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:39230/0 (socket says 172.21.15.102:39230) 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a32000700 1 -- 172.21.15.102:0/48194982 learned_addr learned my addr 172.21.15.102:0/48194982 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.554+0000 7f9a33400700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a3410b360 0x7f9a341a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a32000700 1 -- 172.21.15.102:0/48194982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a3410b360 msgr2=0x7f9a341a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:52.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a32000700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a3410b360 0x7f9a341a2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:52.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a32000700 1 -- 172.21.15.102:0/48194982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a34075e80 msgr2=0x7f9a3419b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:25:52.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a32000700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a34075e80 0x7f9a3419b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:25:52.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a32000700 1 -- 172.21.15.102:0/48194982 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a341a3490 con 0x7f9a34077450 2024-04-26T20:25:52.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a32a00700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a34075e80 0x7f9a3419b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:25:52.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a32000700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a34077450 0x7f9a3419bd70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000ec20 tx=0x7f9a2000ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a2b400700 1 -- 172.21.15.102:0/48194982 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2000cc90 con 0x7f9a34077450 2024-04-26T20:25:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a2b400700 1 -- 172.21.15.102:0/48194982 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a2000ce30 con 0x7f9a34077450 2024-04-26T20:25:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a39600700 1 -- 172.21.15.102:0/48194982 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a341a3780 con 0x7f9a34077450 2024-04-26T20:25:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a2b400700 1 -- 172.21.15.102:0/48194982 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20018880 con 0x7f9a34077450 2024-04-26T20:25:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.555+0000 7f9a39600700 1 -- 172.21.15.102:0/48194982 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a341a3cc0 con 0x7f9a34077450 2024-04-26T20:25:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.557+0000 7f9a39600700 1 -- 172.21.15.102:0/48194982 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a3406a110 con 0x7f9a34077450 2024-04-26T20:25:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.557+0000 7f9a2b400700 1 -- 172.21.15.102:0/48194982 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f9a2001f020 con 0x7f9a34077450 2024-04-26T20:25:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.557+0000 7f9a2b400700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9a18039890 0x7f9a1803bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:25:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.557+0000 7f9a2b400700 1 -- 172.21.15.102:0/48194982 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9a2004cbc0 con 0x7f9a34077450 2024-04-26T20:25:52.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.558+0000 7f9a32a00700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9a18039890 0x7f9a1803bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:25:52.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.558+0000 7f9a32a00700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9a18039890 0x7f9a1803bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9a24009fd0 tx=0x7f9a24019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:25:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.563+0000 7f9a2b400700 1 -- 172.21.15.102:0/48194982 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9a2001fba0 con 0x7f9a34077450 2024-04-26T20:25:52.621 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:52 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:52.239+0000 7f93df541200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T20:25:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:52.805+0000 7f9a39600700 1 -- 172.21.15.102:0/48194982 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9a34063920 con 0x7f9a18039890 2024-04-26T20:25:52.864 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:52 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:52.526+0000 7f0c44246200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T20:25:52.864 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:52 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:52.694+0000 7f0c44246200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T20:25:52.864 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:52 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:52.863+0000 7f0c44246200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T20:25:53.088 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:52 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:52.764+0000 7f93df541200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T20:25:53.089 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:52 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:52.853+0000 7f93df541200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T20:25:53.371 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:53.088+0000 7f93df541200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T20:25:53.371 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:53.217+0000 7f93df541200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T20:25:53.371 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:53.311+0000 7f93df541200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T20:25:53.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:53.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:53.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:53.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T20:25:53.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:25:53.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:25:53.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:53.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: Reconfiguring daemon mgr.x on smithi102 2024-04-26T20:25:53.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:25:53.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:25:53.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:53.438 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:53.145+0000 7f0c44246200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T20:25:53.438 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:53.237+0000 7f0c44246200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T20:25:53.438 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:53.437+0000 7f0c44246200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T20:25:53.438 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.438 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: Reconfiguring daemon mgr.x on smithi102 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:25:53.439 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:53.521 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.521 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: Reconfiguring daemon mgr.x on smithi102 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:25:53.522 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:25:53.523 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:53.725 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:53.423+0000 7f93df541200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T20:25:53.725 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:53.724+0000 7f93df541200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T20:25:53.776 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:53 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:53.627+0000 7f0c44246200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T20:25:54.526 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:54 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:54.221+0000 7f0c44246200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T20:25:54.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:54 smithi112 ceph-mon[32769]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:54.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:54 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:54.619 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:54 smithi102 ceph-mon[28094]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:54.620 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:54 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:54.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:54 smithi130 ceph-mon[32704]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:25:54.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:54 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:54.621 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:54 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:54.310+0000 7f93df541200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T20:25:54.621 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:25:54 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z[34580]: 2024-04-26T20:25:54.520+0000 7f93df541200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T20:25:54.936 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:54 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:54.640+0000 7f0c44246200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T20:25:54.936 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:54 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:54.738+0000 7f0c44246200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T20:25:54.936 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:54 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:54.936+0000 7f0c44246200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T20:25:55.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:55.533+0000 7f9a2b400700 1 -- 172.21.15.102:0/48194982 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mgrmap(e 14) ==== 92603+0+0 (secure 0 0 0) 0x7f9a20007d00 con 0x7f9a34077450 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: Standby manager daemon z started 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T20:25:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:55 smithi102 ceph-mon[28094]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:25:55.776 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:55.475+0000 7f0c44246200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T20:25:55.776 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:55.567+0000 7f0c44246200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: Standby manager daemon z started 2024-04-26T20:25:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T20:25:55.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-mon[32769]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: Standby manager daemon z started 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T20:25:55.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:55 smithi130 ceph-mon[32704]: from='mgr.? 172.21.15.130:0/1020825086' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:25:56.142 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:55.804+0000 7f0c44246200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T20:25:56.143 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:55.935+0000 7f0c44246200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T20:25:56.143 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:56 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:56.028+0000 7f0c44246200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T20:25:56.143 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:56 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:56.142+0000 7f0c44246200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T20:25:56.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:56 smithi102 ceph-mon[28094]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-26T20:25:56.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T20:25:56.776 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:56 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:56.453+0000 7f0c44246200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T20:25:56.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:56 smithi112 ceph-mon[32769]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-26T20:25:56.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T20:25:56.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:56 smithi130 ceph-mon[32704]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-26T20:25:56.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T20:25:57.526 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:57 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:57.043+0000 7f0c44246200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T20:25:57.527 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:25:57 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y[34173]: 2024-04-26T20:25:57.255+0000 7f0c44246200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T20:25:57.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:25:57.553+0000 7f9a2b400700 1 -- 172.21.15.102:0/48194982 <== mon.1 v2:172.21.15.130:3300/0 8 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7f9a20016330 con 0x7f9a34077450 2024-04-26T20:25:57.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:57 smithi130 ceph-mon[32704]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:57.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:57 smithi130 ceph-mon[32704]: Standby manager daemon y started 2024-04-26T20:25:57.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:57 smithi130 ceph-mon[32704]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T20:25:57.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:57 smithi130 ceph-mon[32704]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:25:57.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:57 smithi130 ceph-mon[32704]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T20:25:57.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:57 smithi130 ceph-mon[32704]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:25:57.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:57 smithi102 ceph-mon[28094]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:57.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:57 smithi102 ceph-mon[28094]: Standby manager daemon y started 2024-04-26T20:25:57.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:57 smithi102 ceph-mon[28094]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T20:25:57.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:57 smithi102 ceph-mon[28094]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:25:57.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:57 smithi102 ceph-mon[28094]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T20:25:57.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:57 smithi102 ceph-mon[28094]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:25:58.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:57 smithi112 ceph-mon[32769]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:57 smithi112 ceph-mon[32769]: Standby manager daemon y started 2024-04-26T20:25:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:57 smithi112 ceph-mon[32769]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T20:25:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:57 smithi112 ceph-mon[32769]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T20:25:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:57 smithi112 ceph-mon[32769]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T20:25:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:57 smithi112 ceph-mon[32769]: from='mgr.? 172.21.15.112:0/3457091695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T20:25:58.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:58 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/173569360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00e64c1-d8b2-4ede-a6e5-236cac247fd2"}]: dispatch 2024-04-26T20:25:58.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:58 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00e64c1-d8b2-4ede-a6e5-236cac247fd2"}]: dispatch 2024-04-26T20:25:58.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:58 smithi130 ceph-mon[32704]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-26T20:25:58.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:58 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T20:25:58.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:58 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b00e64c1-d8b2-4ede-a6e5-236cac247fd2"}]': finished 2024-04-26T20:25:58.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:58 smithi130 ceph-mon[32704]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T20:25:58.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:58 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:25:58.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:58 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1564482291' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:25:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:58 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/173569360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00e64c1-d8b2-4ede-a6e5-236cac247fd2"}]: dispatch 2024-04-26T20:25:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:58 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00e64c1-d8b2-4ede-a6e5-236cac247fd2"}]: dispatch 2024-04-26T20:25:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:58 smithi102 ceph-mon[28094]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-26T20:25:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:58 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T20:25:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:58 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b00e64c1-d8b2-4ede-a6e5-236cac247fd2"}]': finished 2024-04-26T20:25:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:58 smithi102 ceph-mon[28094]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T20:25:58.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:58 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:25:58.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:58 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1564482291' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:25:59.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:58 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/173569360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00e64c1-d8b2-4ede-a6e5-236cac247fd2"}]: dispatch 2024-04-26T20:25:59.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:58 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b00e64c1-d8b2-4ede-a6e5-236cac247fd2"}]: dispatch 2024-04-26T20:25:59.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:58 smithi112 ceph-mon[32769]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-26T20:25:59.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:58 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T20:25:59.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:58 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b00e64c1-d8b2-4ede-a6e5-236cac247fd2"}]': finished 2024-04-26T20:25:59.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:58 smithi112 ceph-mon[32769]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T20:25:59.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:58 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:25:59.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:58 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1564482291' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:25:59.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:25:59 smithi130 ceph-mon[32704]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:25:59.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:25:59 smithi102 ceph-mon[28094]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:00.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:25:59 smithi112 ceph-mon[32769]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:01.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:01 smithi130 ceph-mon[32704]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:01.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:01 smithi102 ceph-mon[28094]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:02.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:01 smithi112 ceph-mon[32769]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:03.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:03 smithi130 ceph-mon[32704]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:03.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:03 smithi102 ceph-mon[28094]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:04.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:03 smithi112 ceph-mon[32769]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:05.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:05 smithi130 ceph-mon[32704]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:05.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:05 smithi102 ceph-mon[28094]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:06.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:05 smithi112 ceph-mon[32769]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:07.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:07 smithi102 ceph-mon[28094]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:07.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:07 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T20:26:07.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:07 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:07.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:07 smithi130 ceph-mon[32704]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:07.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:07 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T20:26:07.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:07 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:08.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:07 smithi112 ceph-mon[32769]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:08.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:07 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T20:26:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:07 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:08.865 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:08 smithi102 ceph-mon[28094]: Deploying daemon osd.0 on smithi102 2024-04-26T20:26:08.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:08 smithi130 ceph-mon[32704]: Deploying daemon osd.0 on smithi102 2024-04-26T20:26:09.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:08 smithi112 ceph-mon[32769]: Deploying daemon osd.0 on smithi102 2024-04-26T20:26:09.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:09 smithi102 ceph-mon[28094]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:09.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:09 smithi130 ceph-mon[32704]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:10.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:09 smithi112 ceph-mon[32769]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:11.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:11 smithi130 ceph-mon[32704]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:11.888 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:11 smithi102 ceph-mon[28094]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:12.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:11 smithi112 ceph-mon[32769]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:12.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:12 smithi102 ceph-mon[28094]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:13.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:12 smithi112 ceph-mon[32769]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:13.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:12 smithi130 ceph-mon[32704]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:15.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:15 smithi112 ceph-mon[32769]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:15.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:15 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:15.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:15 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:15.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:15 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:15 smithi130 ceph-mon[32704]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:15 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:15 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:15 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:15 smithi102 ceph-mon[28094]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:17.588 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:17 smithi102 ceph-mon[28094]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:17.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:17 smithi112 ceph-mon[32769]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:17.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:17 smithi130 ceph-mon[32704]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:18.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:18.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:18 smithi102 ceph-mon[28094]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:19.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:18 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:19.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:18 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:19.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:18 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:19.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:18 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:19.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:18 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:19.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:18 smithi112 ceph-mon[32769]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:19.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:18 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:19.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:18 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:19.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:18 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:19.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:18 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:19.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:18 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:19.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:18 smithi130 ceph-mon[32704]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:19.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.815+0000 7f9a2b400700 1 -- 172.21.15.102:0/48194982 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9a34063920 con 0x7f9a18039890 2024-04-26T20:26:19.817 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 0 on host 'smithi102' 2024-04-26T20:26:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.820+0000 7f9a39600700 1 -- 172.21.15.102:0/48194982 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9a18039890 msgr2=0x7f9a1803bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:26:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.820+0000 7f9a39600700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9a18039890 0x7f9a1803bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9a24009fd0 tx=0x7f9a24019040 comp rx=0 tx=0).stop 2024-04-26T20:26:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.820+0000 7f9a39600700 1 -- 172.21.15.102:0/48194982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a34077450 msgr2=0x7f9a3419bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:26:19.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.820+0000 7f9a39600700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a34077450 0x7f9a3419bd70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000ec20 tx=0x7f9a2000ef30 comp rx=0 tx=0).stop 2024-04-26T20:26:19.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.820+0000 7f9a39600700 1 -- 172.21.15.102:0/48194982 shutdown_connections 2024-04-26T20:26:19.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.820+0000 7f9a39600700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9a18039890 0x7f9a1803bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:19.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.820+0000 7f9a39600700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a3410b360 0x7f9a341a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:19.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.820+0000 7f9a39600700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a34075e80 0x7f9a3419b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:19.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.820+0000 7f9a39600700 1 --2- 172.21.15.102:0/48194982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a34077450 0x7f9a3419bd70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:19.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.821+0000 7f9a39600700 1 -- 172.21.15.102:0/48194982 >> 172.21.15.102:0/48194982 conn(0x7f9a340fe9f0 msgr2=0x7f9a34107260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:26:19.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.821+0000 7f9a39600700 1 -- 172.21.15.102:0/48194982 shutdown_connections 2024-04-26T20:26:19.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:26:19.821+0000 7f9a39600700 1 -- 172.21.15.102:0/48194982 wait complete. 2024-04-26T20:26:20.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:19 smithi112 ceph-mon[32769]: from='osd.0 [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T20:26:20.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:19 smithi112 ceph-mon[32769]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T20:26:20.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:19 smithi130 ceph-mon[32704]: from='osd.0 [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T20:26:20.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:19 smithi130 ceph-mon[32704]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T20:26:20.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:19 smithi102 ceph-mon[28094]: from='osd.0 [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T20:26:20.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:19 smithi102 ceph-mon[28094]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T20:26:20.653 DEBUG:teuthology.orchestra.run.smithi102:osd.0> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T20:26:20.659 INFO:tasks.cephadm:Deploying osd.1 on smithi112 with /dev/vg_nvme/lv_4... 2024-04-26T20:26:20.659 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T20:26:20.891 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:20 smithi102 ceph-mon[28094]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T20:26:20.891 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:20 smithi102 ceph-mon[28094]: from='osd.0 [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:26:20.891 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:20 smithi102 ceph-mon[28094]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T20:26:20.891 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:20 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:20.891 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:20 smithi102 ceph-mon[28094]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:26:20.891 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:20 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:20.891 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:20 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:20.891 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:20 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:20.891 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:20 smithi102 ceph-mon[28094]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:20.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:20 smithi102 ceph-mon[28094]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:26:20.916 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:26:21.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:20 smithi112 ceph-mon[32769]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T20:26:21.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:20 smithi112 ceph-mon[32769]: from='osd.0 [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:26:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:20 smithi112 ceph-mon[32769]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T20:26:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:20 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:20 smithi112 ceph-mon[32769]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:26:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:20 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:20 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:20 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:20 smithi112 ceph-mon[32769]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:20 smithi112 ceph-mon[32769]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:26:21.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:20 smithi130 ceph-mon[32704]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T20:26:21.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:20 smithi130 ceph-mon[32704]: from='osd.0 [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:26:21.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:20 smithi130 ceph-mon[32704]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T20:26:21.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:20 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:21.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:20 smithi130 ceph-mon[32704]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:26:21.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:20 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:21.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:20 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:21.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:20 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:21.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:20 smithi130 ceph-mon[32704]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:21.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:20 smithi130 ceph-mon[32704]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:26:22.009 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:21 smithi112 ceph-mon[32769]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T20:26:22.009 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:22.009 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:22.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:21 smithi102 ceph-mon[28094]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T20:26:22.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:22.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:22.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:21 smithi130 ceph-mon[32704]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T20:26:22.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:22.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:22.661 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T20:26:22.661 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T20:26:22.661 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-04-26T20:26:22.661 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-04-26T20:26:22.661 INFO:teuthology.orchestra.run.smithi112.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114413 s, 916 MB/s 2024-04-26T20:26:22.661 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 2024-04-26T20:26:22.661 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-04-26T20:26:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:22 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:26:22.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:22 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:26:22.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:22 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:22.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:22 smithi102 ceph-mon[28094]: osd.0 [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] boot 2024-04-26T20:26:22.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:22 smithi102 ceph-mon[28094]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T20:26:22.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:22 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:22.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:22 smithi102 ceph-mon[28094]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:23.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:22 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:26:23.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:22 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:26:23.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:22 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:23.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:22 smithi112 ceph-mon[32769]: osd.0 [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] boot 2024-04-26T20:26:23.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:22 smithi112 ceph-mon[32769]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T20:26:23.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:22 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:23.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:22 smithi112 ceph-mon[32769]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:23.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:22 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:26:23.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:22 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:26:23.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:23.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:22 smithi130 ceph-mon[32704]: osd.0 [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] boot 2024-04-26T20:26:23.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:22 smithi130 ceph-mon[32704]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T20:26:23.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T20:26:23.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:22 smithi130 ceph-mon[32704]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T20:26:23.268 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_4 2024-04-26T20:26:23.503 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:26:23.975 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:23 smithi112 ceph-mon[32769]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T20:26:24.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:23 smithi130 ceph-mon[32704]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T20:26:24.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:23 smithi102 ceph-mon[28094]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T20:26:24.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:24 smithi102 ceph-mon[28094]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.965+0000 7f22fac93700 1 -- 172.21.15.112:0/1620157101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22f4103920 msgr2=0x7f22f4103d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.965+0000 7f22fac93700 1 --2- 172.21.15.112:0/1620157101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22f4103920 0x7f22f4103d40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f22e8009990 tx=0x7f22e8009ca0 comp rx=0 tx=0).stop 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.965+0000 7f22fac93700 1 -- 172.21.15.112:0/1620157101 shutdown_connections 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.965+0000 7f22fac93700 1 --2- 172.21.15.112:0/1620157101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22f4103000 0x7f22f410a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.965+0000 7f22fac93700 1 --2- 172.21.15.112:0/1620157101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22f4103920 0x7f22f4103d40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.965+0000 7f22fac93700 1 --2- 172.21.15.112:0/1620157101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22f4102660 0x7f22f4102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.965+0000 7f22fac93700 1 -- 172.21.15.112:0/1620157101 >> 172.21.15.112:0/1620157101 conn(0x7f22f40fd090 msgr2=0x7f22f40ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.965+0000 7f22fac93700 1 -- 172.21.15.112:0/1620157101 shutdown_connections 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.965+0000 7f22fac93700 1 -- 172.21.15.112:0/1620157101 wait complete. 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22fac93700 1 Processor -- start 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22fac93700 1 -- start start 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22fac93700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22f4102660 0x7f22f4198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:26:24.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22fac93700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22f4103000 0x7f22f4198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22fac93700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22f4103920 0x7f22f419fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22fac93700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410def0 con 0x7f22f4103920 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22fac93700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410dd70 con 0x7f22f4103000 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22fac93700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410e070 con 0x7f22f4102660 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22f3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22f4103000 0x7f22f4198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22f3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22f4103000 0x7f22f4198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:36756/0 (socket says 172.21.15.112:36756) 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.966+0000 7f22f3fff700 1 -- 172.21.15.112:0/2090781829 learned_addr learned my addr 172.21.15.112:0/2090781829 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f8a2f700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22f4102660 0x7f22f4198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f3fff700 1 -- 172.21.15.112:0/2090781829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22f4102660 msgr2=0x7f22f4198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f3fff700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22f4102660 0x7f22f4198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f3fff700 1 -- 172.21.15.112:0/2090781829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22f4103920 msgr2=0x7f22f419fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f3fff700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22f4103920 0x7f22f419fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f3fff700 1 -- 172.21.15.112:0/2090781829 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f41a01c0 con 0x7f22f4103000 2024-04-26T20:26:24.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f8a2f700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22f4102660 0x7f22f4198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:26:24.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f3fff700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22f4103000 0x7f22f4198aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f22e000d9e0 tx=0x7f22e000dda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:26:24.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f1ffb700 1 -- 172.21.15.112:0/2090781829 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e001b070 con 0x7f22f4103000 2024-04-26T20:26:24.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f1ffb700 1 -- 172.21.15.112:0/2090781829 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22e000ec50 con 0x7f22f4103000 2024-04-26T20:26:24.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22fac93700 1 -- 172.21.15.112:0/2090781829 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f41015b0 con 0x7f22f4103000 2024-04-26T20:26:24.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22f1ffb700 1 -- 172.21.15.112:0/2090781829 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e0014630 con 0x7f22f4103000 2024-04-26T20:26:24.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.967+0000 7f22fac93700 1 -- 172.21.15.112:0/2090781829 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f4101bc0 con 0x7f22f4103000 2024-04-26T20:26:24.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.968+0000 7f22fac93700 1 -- 172.21.15.112:0/2090781829 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f406a110 con 0x7f22f4103000 2024-04-26T20:26:24.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.969+0000 7f22f1ffb700 1 -- 172.21.15.112:0/2090781829 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7f22e001e070 con 0x7f22f4103000 2024-04-26T20:26:24.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.970+0000 7f22f1ffb700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f22e40a3c30 0x7f22e40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:26:24.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.970+0000 7f22f1ffb700 1 -- 172.21.15.112:0/2090781829 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f22e00cfdd0 con 0x7f22f4103000 2024-04-26T20:26:24.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.970+0000 7f22f8a2f700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f22e40a3c30 0x7f22e40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:26:24.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.971+0000 7f22f8a2f700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f22e40a3c30 0x7f22e40a60f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f22e800b850 tx=0x7f22e801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:26:24.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:24.974+0000 7f22f1ffb700 1 -- 172.21.15.112:0/2090781829 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f22e009e2b0 con 0x7f22f4103000 2024-04-26T20:26:25.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:24 smithi112 ceph-mon[32769]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:24 smithi130 ceph-mon[32704]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:25.180 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:25.179+0000 7f22fac93700 1 -- 172.21.15.112:0/2090781829 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f22f4101e80 con 0x7f22e40a3c30 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi102 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi102 to 17089M 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.017 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:26.018 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:26.018 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:26:26.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:26:26.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:26:26.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:26.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi102 2024-04-26T20:26:26.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:26:26.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi102 to 17089M 2024-04-26T20:26:26.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:26.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:26.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:26:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:26:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:26:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi102 2024-04-26T20:26:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:26:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi102 to 17089M 2024-04-26T20:26:26.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:26.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:26.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:26.887 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:26 smithi112 ceph-mon[32769]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:27.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:26 smithi130 ceph-mon[32704]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:27.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:26 smithi102 ceph-mon[28094]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:27 smithi112 ceph-mon[32769]: from='client.? 172.21.15.112:0/1796359848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e10d054b-2732-48ef-a2ba-4137acda22bb"}]: dispatch 2024-04-26T20:26:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:27 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e10d054b-2732-48ef-a2ba-4137acda22bb"}]: dispatch 2024-04-26T20:26:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:27 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e10d054b-2732-48ef-a2ba-4137acda22bb"}]': finished 2024-04-26T20:26:28.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:27 smithi130 ceph-mon[32704]: from='client.? 172.21.15.112:0/1796359848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e10d054b-2732-48ef-a2ba-4137acda22bb"}]: dispatch 2024-04-26T20:26:28.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:27 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e10d054b-2732-48ef-a2ba-4137acda22bb"}]: dispatch 2024-04-26T20:26:28.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:27 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e10d054b-2732-48ef-a2ba-4137acda22bb"}]': finished 2024-04-26T20:26:28.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:27 smithi102 ceph-mon[28094]: from='client.? 172.21.15.112:0/1796359848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e10d054b-2732-48ef-a2ba-4137acda22bb"}]: dispatch 2024-04-26T20:26:28.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:27 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e10d054b-2732-48ef-a2ba-4137acda22bb"}]: dispatch 2024-04-26T20:26:28.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:27 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e10d054b-2732-48ef-a2ba-4137acda22bb"}]': finished 2024-04-26T20:26:29.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:28 smithi130 ceph-mon[32704]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T20:26:29.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:29.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:28 smithi130 ceph-mon[32704]: from='client.? 172.21.15.112:0/3401305755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:26:29.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:28 smithi130 ceph-mon[32704]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:29.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:28 smithi102 ceph-mon[28094]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T20:26:29.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:29.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:28 smithi102 ceph-mon[28094]: from='client.? 172.21.15.112:0/3401305755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:26:29.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:28 smithi102 ceph-mon[28094]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:29.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:28 smithi112 ceph-mon[32769]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T20:26:29.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:28 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:29.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:28 smithi112 ceph-mon[32769]: from='client.? 172.21.15.112:0/3401305755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:26:29.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:28 smithi112 ceph-mon[32769]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:31.677 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:31 smithi112 ceph-mon[32769]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:31.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:31 smithi130 ceph-mon[32704]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:31.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:31 smithi102 ceph-mon[28094]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:33.688 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:33 smithi112 ceph-mon[32769]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:33 smithi130 ceph-mon[32704]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:33.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:33 smithi102 ceph-mon[28094]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:35.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:35 smithi102 ceph-mon[28094]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:35.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T20:26:35.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:35.692 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:35 smithi112 ceph-mon[32769]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:35.692 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T20:26:35.692 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:35.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:35 smithi130 ceph-mon[32704]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:35.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T20:26:35.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:36.605 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:36 smithi112 ceph-mon[32769]: Deploying daemon osd.1 on smithi112 2024-04-26T20:26:36.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:36 smithi130 ceph-mon[32704]: Deploying daemon osd.1 on smithi112 2024-04-26T20:26:36.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:36 smithi102 ceph-mon[28094]: Deploying daemon osd.1 on smithi112 2024-04-26T20:26:37.502 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:37 smithi112 ceph-mon[32769]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:37.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:37 smithi130 ceph-mon[32704]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:37.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:37 smithi102 ceph-mon[28094]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:39.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:39 smithi112 ceph-mon[32769]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:39.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:39 smithi130 ceph-mon[32704]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:39.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:39 smithi102 ceph-mon[28094]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:41.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:41 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:41.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:41 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:41.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:41 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:41.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:41 smithi130 ceph-mon[32704]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:41.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:41 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:41.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:41 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:41.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:41 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:41.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:41 smithi102 ceph-mon[28094]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:41.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:41 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:41.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:41 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:41.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:41 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:41.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:41 smithi112 ceph-mon[32769]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:43.727 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:43 smithi112 ceph-mon[32769]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:43.727 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:43.727 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:43.727 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:43.727 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:43.727 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:43.819 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 1 on host 'smithi112' 2024-04-26T20:26:43.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.819+0000 7f22f1ffb700 1 -- 172.21.15.112:0/2090781829 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f22f4101e80 con 0x7f22e40a3c30 2024-04-26T20:26:43.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 -- 172.21.15.112:0/2090781829 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f22e40a3c30 msgr2=0x7f22e40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f22e40a3c30 0x7f22e40a60f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f22e800b850 tx=0x7f22e801a040 comp rx=0 tx=0).stop 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 -- 172.21.15.112:0/2090781829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22f4103000 msgr2=0x7f22f4198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22f4103000 0x7f22f4198aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f22e000d9e0 tx=0x7f22e000dda0 comp rx=0 tx=0).stop 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 -- 172.21.15.112:0/2090781829 shutdown_connections 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f22e40a3c30 0x7f22e40a60f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22f4103920 0x7f22f419fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22f4102660 0x7f22f4198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 --2- 172.21.15.112:0/2090781829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22f4103000 0x7f22f4198aa0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 -- 172.21.15.112:0/2090781829 >> 172.21.15.112:0/2090781829 conn(0x7f22f40fd090 msgr2=0x7f22f41069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 -- 172.21.15.112:0/2090781829 shutdown_connections 2024-04-26T20:26:43.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:26:43.821+0000 7f22fac93700 1 -- 172.21.15.112:0/2090781829 wait complete. 2024-04-26T20:26:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:43 smithi130 ceph-mon[32704]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:43.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:43 smithi102 ceph-mon[28094]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:43.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:43.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:44.492 DEBUG:teuthology.orchestra.run.smithi112:osd.1> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T20:26:44.494 INFO:tasks.cephadm:Deploying osd.2 on smithi130 with /dev/vg_nvme/lv_4... 2024-04-26T20:26:44.494 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T20:26:44.740 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:26:44.861 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:44.861 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:44.861 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:44.861 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:44 smithi130 ceph-mon[32704]: from='osd.1 [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:26:44.861 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:44 smithi130 ceph-mon[32704]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:26:44.862 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:44 smithi130 ceph-mon[32704]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:45.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:45.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:45.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:45.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:44 smithi112 ceph-mon[32769]: from='osd.1 [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:26:45.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:44 smithi112 ceph-mon[32769]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:26:45.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:44 smithi112 ceph-mon[32769]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:45.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:45.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:45.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:26:45.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:44 smithi102 ceph-mon[28094]: from='osd.1 [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:26:45.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:44 smithi102 ceph-mon[28094]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T20:26:45.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:44 smithi102 ceph-mon[28094]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:45 smithi130 ceph-mon[32704]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T20:26:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:45 smithi130 ceph-mon[32704]: from='osd.1 [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:26:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:45 smithi130 ceph-mon[32704]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T20:26:46.083 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:45 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:46.084 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:45 smithi130 ceph-mon[32704]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:26:46.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:45 smithi102 ceph-mon[28094]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T20:26:46.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:45 smithi102 ceph-mon[28094]: from='osd.1 [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:26:46.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:45 smithi102 ceph-mon[28094]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T20:26:46.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:45 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:46.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:45 smithi102 ceph-mon[28094]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:26:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:45 smithi112 ceph-mon[32769]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T20:26:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:45 smithi112 ceph-mon[32769]: from='osd.1 [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:26:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:45 smithi112 ceph-mon[32769]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T20:26:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:45 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:45 smithi112 ceph-mon[32769]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:26:46.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:26:45 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1[37369]: 2024-04-26T20:26:45.791+0000 7f1ff7a7d700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T20:26:46.278 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:26:45 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1[37369]: 2024-04-26T20:26:45.795+0000 7f1ff3870700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:26:46.753 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T20:26:46.754 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T20:26:46.754 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-26T20:26:46.754 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-26T20:26:46.754 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129567 s, 809 MB/s 2024-04-26T20:26:46.754 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-26T20:26:46.976 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:46 smithi112 ceph-mon[32769]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:26:46.976 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:46 smithi112 ceph-mon[32769]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T20:26:46.976 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:46.976 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:46 smithi112 ceph-mon[32769]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:46.976 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:47.074 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:46 smithi130 ceph-mon[32704]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:26:47.074 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:46 smithi130 ceph-mon[32704]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T20:26:47.074 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:47.074 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:46 smithi130 ceph-mon[32704]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:47.074 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:47.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:46 smithi102 ceph-mon[28094]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:26:47.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:46 smithi102 ceph-mon[28094]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T20:26:47.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:47.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:46 smithi102 ceph-mon[28094]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T20:26:47.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:47.369 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi130:vg_nvme/lv_4 2024-04-26T20:26:47.600 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:26:48.019 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:26:48.019 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:26:48.019 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: osd.1 [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] boot 2024-04-26T20:26:48.019 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T20:26:48.019 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:48.020 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi112 2024-04-26T20:26:48.020 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.020 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.020 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:26:48.020 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi112 to 17088M 2024-04-26T20:26:48.020 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.020 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:48.020 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:48.020 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:26:48.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:26:48.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: osd.1 [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] boot 2024-04-26T20:26:48.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T20:26:48.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:48.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi112 2024-04-26T20:26:48.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:26:48.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi112 to 17088M 2024-04-26T20:26:48.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:48.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:48.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:26:48.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:26:48.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: osd.1 [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] boot 2024-04-26T20:26:48.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T20:26:48.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T20:26:48.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi112 2024-04-26T20:26:48.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:26:48.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi112 to 17088M 2024-04-26T20:26:48.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:48.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:26:48.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:26:48.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:48 smithi112 ceph-mon[32769]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T20:26:48.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:48 smithi112 ceph-mon[32769]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:48.954 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.951+0000 7f13b24c0700 1 -- 172.21.15.130:0/3972898885 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13ac103f20 msgr2=0x7f13ac10a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:26:48.954 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.951+0000 7f13b24c0700 1 --2- 172.21.15.130:0/3972898885 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13ac103f20 0x7f13ac10a7d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f13a400b0a0 tx=0x7f13a400b3b0 comp rx=0 tx=0).stop 2024-04-26T20:26:48.954 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.951+0000 7f13b24c0700 1 -- 172.21.15.130:0/3972898885 shutdown_connections 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.951+0000 7f13b24c0700 1 --2- 172.21.15.130:0/3972898885 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13ac103f20 0x7f13ac10a7d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.951+0000 7f13b24c0700 1 --2- 172.21.15.130:0/3972898885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac102380 0x7f13ac1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.951+0000 7f13b24c0700 1 --2- 172.21.15.130:0/3972898885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac103580 0x7f13ac1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.951+0000 7f13b24c0700 1 -- 172.21.15.130:0/3972898885 >> 172.21.15.130:0/3972898885 conn(0x7f13ac0fd920 msgr2=0x7f13ac0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.951+0000 7f13b24c0700 1 -- 172.21.15.130:0/3972898885 shutdown_connections 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.951+0000 7f13b24c0700 1 -- 172.21.15.130:0/3972898885 wait complete. 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13b24c0700 1 Processor -- start 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13b24c0700 1 -- start start 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13b24c0700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13ac102380 0x7f13ac1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13b24c0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac103580 0x7f13ac1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13b24c0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac103f20 0x7f13ac19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:26:48.955 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13b24c0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac10deb0 con 0x7f13ac103f20 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13b24c0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac10dd30 con 0x7f13ac103580 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13b24c0700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac10e030 con 0x7f13ac102380 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13abfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13ac102380 0x7f13ac1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13abfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13ac102380 0x7f13ac1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:43374/0 (socket says 172.21.15.130:43374) 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13abfff700 1 -- 172.21.15.130:0/1106128730 learned_addr learned my addr 172.21.15.130:0/1106128730 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.952+0000 7f13ab7fe700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac103580 0x7f13ac1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13abfff700 1 -- 172.21.15.130:0/1106128730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac103580 msgr2=0x7f13ac1989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13abfff700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac103580 0x7f13ac1989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13abfff700 1 -- 172.21.15.130:0/1106128730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac103f20 msgr2=0x7f13ac19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13abfff700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac103f20 0x7f13ac19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13abfff700 1 -- 172.21.15.130:0/1106128730 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac06bcb0 con 0x7f13ac102380 2024-04-26T20:26:48.956 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13abfff700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13ac102380 0x7f13ac1984a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f13a000b500 tx=0x7f13a000b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:26:48.957 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13a97fa700 1 -- 172.21.15.130:0/1106128730 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0015020 con 0x7f13ac102380 2024-04-26T20:26:48.957 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13a97fa700 1 -- 172.21.15.130:0/1106128730 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a0011ce0 con 0x7f13ac102380 2024-04-26T20:26:48.957 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13b24c0700 1 -- 172.21.15.130:0/1106128730 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ac06bfa0 con 0x7f13ac102380 2024-04-26T20:26:48.957 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13a97fa700 1 -- 172.21.15.130:0/1106128730 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0010d20 con 0x7f13ac102380 2024-04-26T20:26:48.957 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.953+0000 7f13b24c0700 1 -- 172.21.15.130:0/1106128730 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ac06c560 con 0x7f13ac102380 2024-04-26T20:26:48.958 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.954+0000 7f13b24c0700 1 -- 172.21.15.130:0/1106128730 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13ac06a110 con 0x7f13ac102380 2024-04-26T20:26:48.958 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.955+0000 7f13a97fa700 1 -- 172.21.15.130:0/1106128730 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7f13a0018020 con 0x7f13ac102380 2024-04-26T20:26:48.958 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.955+0000 7f13a97fa700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f13940a3bb0 0x7f13940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:26:48.958 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.956+0000 7f13a97fa700 1 -- 172.21.15.130:0/1106128730 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f13a00ce2c0 con 0x7f13ac102380 2024-04-26T20:26:48.958 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.956+0000 7f13ab7fe700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f13940a3bb0 0x7f13940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:26:48.958 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.957+0000 7f13ab7fe700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f13940a3bb0 0x7f13940a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f139c004630 tx=0x7f139c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:26:48.961 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:48.960+0000 7f13a97fa700 1 -- 172.21.15.130:0/1106128730 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f13a001f020 con 0x7f13ac102380 2024-04-26T20:26:49.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:48 smithi130 ceph-mon[32704]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T20:26:49.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:48 smithi130 ceph-mon[32704]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:49.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:48 smithi102 ceph-mon[28094]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T20:26:49.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:48 smithi102 ceph-mon[28094]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:49.167 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:26:49.167+0000 7f13b24c0700 1 -- 172.21.15.130:0/1106128730 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f13ac02e500 con 0x7f13940a3bb0 2024-04-26T20:26:50.063 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:49 smithi130 ceph-mon[32704]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:26:50.063 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:49 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:26:50.063 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:49 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:26:50.063 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:49 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:50.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:49 smithi102 ceph-mon[28094]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:26:50.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:49 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:26:50.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:49 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:26:50.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:49 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:50.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:49 smithi112 ceph-mon[32769]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:26:50.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:49 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:26:50.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:49 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:26:50.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:49 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:50.888 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:50 smithi130 ceph-mon[32704]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:51.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:50 smithi102 ceph-mon[28094]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:51.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:50 smithi112 ceph-mon[32769]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:52.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:51 smithi130 ceph-mon[32704]: from='client.? 172.21.15.130:0/599106364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d090636-b81e-404e-9b1d-d1919ab42ba3"}]: dispatch 2024-04-26T20:26:52.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:51 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d090636-b81e-404e-9b1d-d1919ab42ba3"}]: dispatch 2024-04-26T20:26:52.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:51 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d090636-b81e-404e-9b1d-d1919ab42ba3"}]': finished 2024-04-26T20:26:52.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:51 smithi130 ceph-mon[32704]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T20:26:52.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:26:52.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:51 smithi102 ceph-mon[28094]: from='client.? 172.21.15.130:0/599106364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d090636-b81e-404e-9b1d-d1919ab42ba3"}]: dispatch 2024-04-26T20:26:52.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:51 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d090636-b81e-404e-9b1d-d1919ab42ba3"}]: dispatch 2024-04-26T20:26:52.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:51 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d090636-b81e-404e-9b1d-d1919ab42ba3"}]': finished 2024-04-26T20:26:52.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:51 smithi102 ceph-mon[28094]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T20:26:52.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:26:52.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:51 smithi112 ceph-mon[32769]: from='client.? 172.21.15.130:0/599106364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d090636-b81e-404e-9b1d-d1919ab42ba3"}]: dispatch 2024-04-26T20:26:52.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:51 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d090636-b81e-404e-9b1d-d1919ab42ba3"}]: dispatch 2024-04-26T20:26:52.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:51 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d090636-b81e-404e-9b1d-d1919ab42ba3"}]': finished 2024-04-26T20:26:52.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:51 smithi112 ceph-mon[32769]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T20:26:52.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:26:53.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:52 smithi130 ceph-mon[32704]: from='client.? 172.21.15.130:0/698567162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:26:53.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:52 smithi130 ceph-mon[32704]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:53.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:52 smithi102 ceph-mon[28094]: from='client.? 172.21.15.130:0/698567162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:26:53.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:52 smithi102 ceph-mon[28094]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:53.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:52 smithi112 ceph-mon[32769]: from='client.? 172.21.15.130:0/698567162' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:26:53.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:52 smithi112 ceph-mon[32769]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:55.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:55 smithi130 ceph-mon[32704]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:55.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:55 smithi112 ceph-mon[32769]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:55.851 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:55 smithi102 ceph-mon[28094]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:57.737 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:57 smithi130 ceph-mon[32704]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:57.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:57 smithi112 ceph-mon[32769]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:57.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:57 smithi102 ceph-mon[28094]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:59.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:59 smithi130 ceph-mon[32704]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:59.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:59 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T20:26:59.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:26:59 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:59.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:59 smithi112 ceph-mon[32769]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:59.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:59 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T20:26:59.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:26:59 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:26:59.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:59 smithi102 ceph-mon[28094]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:26:59.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:59 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T20:26:59.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:26:59 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:00.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:00 smithi112 ceph-mon[32769]: Deploying daemon osd.2 on smithi130 2024-04-26T20:27:00.800 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:00 smithi130 ceph-mon[32704]: Deploying daemon osd.2 on smithi130 2024-04-26T20:27:00.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:00 smithi102 ceph-mon[28094]: Deploying daemon osd.2 on smithi130 2024-04-26T20:27:01.760 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:01 smithi130 ceph-mon[32704]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:01.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:01 smithi112 ceph-mon[32769]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:01.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:01 smithi102 ceph-mon[28094]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:03.735 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:03 smithi130 ceph-mon[32704]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:03.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:03 smithi112 ceph-mon[32769]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:03.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:03 smithi102 ceph-mon[28094]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:05.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:05.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:05.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:05.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:05 smithi112 ceph-mon[32769]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:05.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:05.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:05.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:05.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:05 smithi130 ceph-mon[32704]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:05.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:05.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:05.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:05.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:05 smithi102 ceph-mon[28094]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:07.498 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 2 on host 'smithi130' 2024-04-26T20:27:07.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.496+0000 7f13a97fa700 1 -- 172.21.15.130:0/1106128730 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f13ac02e500 con 0x7f13940a3bb0 2024-04-26T20:27:07.500 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 -- 172.21.15.130:0/1106128730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f13940a3bb0 msgr2=0x7f13940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:07.500 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f13940a3bb0 0x7f13940a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f139c004630 tx=0x7f139c00b3c0 comp rx=0 tx=0).stop 2024-04-26T20:27:07.500 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 -- 172.21.15.130:0/1106128730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13ac102380 msgr2=0x7f13ac1984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:07.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13ac102380 0x7f13ac1984a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f13a000b500 tx=0x7f13a000b810 comp rx=0 tx=0).stop 2024-04-26T20:27:07.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 -- 172.21.15.130:0/1106128730 shutdown_connections 2024-04-26T20:27:07.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f13940a3bb0 0x7f13940a6070 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:07.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13ac103f20 0x7f13ac19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:07.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13ac102380 0x7f13ac1984a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:07.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 --2- 172.21.15.130:0/1106128730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13ac103580 0x7f13ac1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:07.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 -- 172.21.15.130:0/1106128730 >> 172.21.15.130:0/1106128730 conn(0x7f13ac0fd920 msgr2=0x7f13ac106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:07.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 -- 172.21.15.130:0/1106128730 shutdown_connections 2024-04-26T20:27:07.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:27:07.499+0000 7f13b24c0700 1 -- 172.21.15.130:0/1106128730 wait complete. 2024-04-26T20:27:07.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:07 smithi112 ceph-mon[32769]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:07.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:07 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:07.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:07 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:07.779 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:07 smithi130 ceph-mon[32704]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:07.779 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:07 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:07.779 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:07 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:07.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:07 smithi102 ceph-mon[28094]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:07.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:07 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:07.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:07 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:08.056 DEBUG:teuthology.orchestra.run.smithi130:osd.2> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T20:27:08.058 INFO:tasks.cephadm:Deploying osd.3 on smithi102 with /dev/vg_nvme/lv_3... 2024-04-26T20:27:08.058 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T20:27:08.494 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:27:08.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:08 smithi112 ceph-mon[32769]: from='osd.2 [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:27:08.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:08 smithi112 ceph-mon[32769]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:27:08.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:08 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:08.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:08 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:08.797 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:08 smithi130 ceph-mon[32704]: from='osd.2 [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:27:08.797 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:08 smithi130 ceph-mon[32704]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:27:08.797 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:08 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:08.798 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:08 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:08.809 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:08 smithi102 ceph-mon[28094]: from='osd.2 [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:27:08.809 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:08 smithi102 ceph-mon[28094]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T20:27:08.809 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:08 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:08.809 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:08 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:09.729 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:09 smithi130 ceph-mon[32704]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:09.729 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:09 smithi130 ceph-mon[32704]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T20:27:09.729 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:09 smithi130 ceph-mon[32704]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T20:27:09.729 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:09 smithi130 ceph-mon[32704]: from='osd.2 [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:27:09.729 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:09 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:09.729 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:09 smithi130 ceph-mon[32704]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:27:09.729 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:09 smithi130 ceph-mon[32704]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:27:09.730 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:27:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2[37838]: 2024-04-26T20:27:09.492+0000 7f0511f60700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T20:27:09.730 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:27:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2[37838]: 2024-04-26T20:27:09.500+0000 7f050dd53700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:27:09.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:09 smithi112 ceph-mon[32769]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:09.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:09 smithi112 ceph-mon[32769]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T20:27:09.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:09 smithi112 ceph-mon[32769]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T20:27:09.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:09 smithi112 ceph-mon[32769]: from='osd.2 [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:27:09.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:09 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:09.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:09 smithi112 ceph-mon[32769]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:27:09.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:09 smithi112 ceph-mon[32769]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:27:09.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:09 smithi102 ceph-mon[28094]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:09.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:09 smithi102 ceph-mon[28094]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T20:27:09.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:09 smithi102 ceph-mon[28094]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T20:27:09.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:09 smithi102 ceph-mon[28094]: from='osd.2 [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:27:09.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:09.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:09 smithi102 ceph-mon[28094]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:27:09.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:09 smithi102 ceph-mon[28094]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:27:10.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:10 smithi112 ceph-mon[32769]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T20:27:10.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:10 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:10.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:10 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:10.813 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:10 smithi130 ceph-mon[32704]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T20:27:10.813 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:10 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:10.813 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:10 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:10.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:10 smithi102 ceph-mon[28094]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T20:27:10.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:10 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:10.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:10 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:11.336 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T20:27:11.336 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T20:27:11.336 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-04-26T20:27:11.336 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-04-26T20:27:11.336 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0315024 s, 333 MB/s 2024-04-26T20:27:11.336 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-04-26T20:27:11.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:11 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:27:11.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:11 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:27:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:11 smithi102 ceph-mon[28094]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:11 smithi102 ceph-mon[28094]: osd.2 [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] boot 2024-04-26T20:27:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:11 smithi102 ceph-mon[28094]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T20:27:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:11 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:27:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:11 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:27:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:11 smithi112 ceph-mon[32769]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:11 smithi112 ceph-mon[32769]: osd.2 [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] boot 2024-04-26T20:27:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:11 smithi112 ceph-mon[32769]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T20:27:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:11.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:11 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:27:11.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:11 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:27:11.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:11 smithi130 ceph-mon[32704]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T20:27:11.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:11 smithi130 ceph-mon[32704]: osd.2 [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] boot 2024-04-26T20:27:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:11 smithi130 ceph-mon[32704]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T20:27:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T20:27:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:12.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_3 2024-04-26T20:27:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:12.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:12.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:12.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.840 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:12.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T20:27:12.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T20:27:12.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:27:12.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:13.521 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47160]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:27:13.521 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47160]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:13.521 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47160]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:13.521 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47160]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:13.521 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47163]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:27:13.521 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47163]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:13.521 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47163]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:13.521 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47163]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:13.592 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41057]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:27:13.592 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41057]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:13.592 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41057]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:13.592 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41057]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:13.592 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41060]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:27:13.592 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41060]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:13.592 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41060]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:13.592 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41060]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:13.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47169]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:27:13.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47169]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:13.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47169]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47169]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47172]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47172]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47172]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 sudo[47172]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi130 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi130 to 17088M 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:13.861 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:13.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:27:13.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:27:13.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:27:13.871 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41906]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:27:13.872 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41906]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:13.872 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41906]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:13.872 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41906]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:13.872 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41909]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:27:13.872 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41909]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:13.872 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41909]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:13.872 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41909]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:13.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi130 2024-04-26T20:27:13.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi130 to 17088M 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41912]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41912]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41912]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:13.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41912]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:13.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41915]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:27:13.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41915]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:13.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41915]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:13.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:13 smithi130 sudo[41915]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi130 2024-04-26T20:27:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi130 to 17088M 2024-04-26T20:27:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:27:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T20:27:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:27:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41063]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41063]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41063]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41063]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41066]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41066]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T20:27:14.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41066]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T20:27:14.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:13 smithi112 sudo[41066]: pam_unix(sudo:session): session closed for user root 2024-04-26T20:27:14.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.716+0000 7f9449800700 1 -- 172.21.15.102:0/171240803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f944410abb0 msgr2=0x7f944410b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.716+0000 7f9449800700 1 --2- 172.21.15.102:0/171240803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f944410abb0 0x7f944410b010 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f94300099c0 tx=0x7f9430009cd0 comp rx=0 tx=0).stop 2024-04-26T20:27:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.716+0000 7f9449800700 1 -- 172.21.15.102:0/171240803 shutdown_connections 2024-04-26T20:27:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.716+0000 7f9449800700 1 --2- 172.21.15.102:0/171240803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f944410b550 0x7f9444111e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.716+0000 7f9449800700 1 --2- 172.21.15.102:0/171240803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94441099b0 0x7f9444109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.716+0000 7f9449800700 1 --2- 172.21.15.102:0/171240803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f944410abb0 0x7f944410b010 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.716+0000 7f9449800700 1 -- 172.21.15.102:0/171240803 >> 172.21.15.102:0/171240803 conn(0x7f9444078980 msgr2=0x7f944407adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.716+0000 7f9449800700 1 -- 172.21.15.102:0/171240803 shutdown_connections 2024-04-26T20:27:14.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.716+0000 7f9449800700 1 -- 172.21.15.102:0/171240803 wait complete. 2024-04-26T20:27:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.717+0000 7f9449800700 1 Processor -- start 2024-04-26T20:27:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.717+0000 7f9449800700 1 -- start start 2024-04-26T20:27:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.717+0000 7f9449800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94441099b0 0x7f944419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.717+0000 7f9449800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f944410abb0 0x7f944419fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.717+0000 7f9449800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f944410b550 0x7f94441a7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9449800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9444115470 con 0x7f94441099b0 2024-04-26T20:27:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9449800700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f94441152f0 con 0x7f944410abb0 2024-04-26T20:27:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9449800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f94441155f0 con 0x7f944410b550 2024-04-26T20:27:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9443e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94441099b0 0x7f944419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9443e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94441099b0 0x7f944419fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56108/0 (socket says 172.21.15.102:56108) 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9443e00700 1 -- 172.21.15.102:0/2938070937 learned_addr learned my addr 172.21.15.102:0/2938070937 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9443400700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f944410abb0 0x7f944419fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9448e00700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f944410b550 0x7f94441a7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9443e00700 1 -- 172.21.15.102:0/2938070937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f944410b550 msgr2=0x7f94441a7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9443e00700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f944410b550 0x7f94441a7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9443e00700 1 -- 172.21.15.102:0/2938070937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f944410abb0 msgr2=0x7f944419fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9443e00700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f944410abb0 0x7f944419fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9443e00700 1 -- 172.21.15.102:0/2938070937 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94441a7710 con 0x7f94441099b0 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.718+0000 7f9448e00700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f944410b550 0x7f94441a7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.719+0000 7f9443e00700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94441099b0 0x7f944419fab0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f943400ea80 tx=0x7f943400ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.719+0000 7f9440c00700 1 -- 172.21.15.102:0/2938070937 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943400cbd0 con 0x7f94441099b0 2024-04-26T20:27:14.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.719+0000 7f9449800700 1 -- 172.21.15.102:0/2938070937 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94441a7a00 con 0x7f94441099b0 2024-04-26T20:27:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.719+0000 7f9440c00700 1 -- 172.21.15.102:0/2938070937 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f943400cd70 con 0x7f94441099b0 2024-04-26T20:27:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.719+0000 7f9440c00700 1 -- 172.21.15.102:0/2938070937 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94340186c0 con 0x7f94441099b0 2024-04-26T20:27:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.719+0000 7f9449800700 1 -- 172.21.15.102:0/2938070937 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94441a7f40 con 0x7f94441099b0 2024-04-26T20:27:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.721+0000 7f9449800700 1 -- 172.21.15.102:0/2938070937 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f944406a110 con 0x7f94441099b0 2024-04-26T20:27:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.723+0000 7f9440c00700 1 -- 172.21.15.102:0/2938070937 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f943401f020 con 0x7f94441099b0 2024-04-26T20:27:14.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.724+0000 7f9440c00700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94280a3c40 0x7f94280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:14.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.724+0000 7f9440c00700 1 -- 172.21.15.102:0/2938070937 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f94340cf110 con 0x7f94441099b0 2024-04-26T20:27:14.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.725+0000 7f9443400700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94280a3c40 0x7f94280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:14.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.725+0000 7f9443400700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94280a3c40 0x7f94280a6100 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9430000c00 tx=0x7f9430005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:14.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.729+0000 7f9440c00700 1 -- 172.21.15.102:0/2938070937 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f943409ce20 con 0x7f94441099b0 2024-04-26T20:27:14.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:27:14.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:27:14.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:27:14.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:27:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:27:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:27:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:27:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:27:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T20:27:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:27:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:27:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:27:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:14 smithi130 ceph-mon[32704]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:27:14.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:27:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:27:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:27:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:27:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:27:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:27:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:27:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:27:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T20:27:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:27:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:27:14.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:27:14.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:14 smithi102 ceph-mon[28094]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:27:14.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:14.966+0000 7f9449800700 1 -- 172.21.15.102:0/2938070937 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f9444063860 con 0x7f94280a3c40 2024-04-26T20:27:15.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:27:15.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:27:15.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:27:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:27:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:27:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:27:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:27:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T20:27:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T20:27:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T20:27:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T20:27:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T20:27:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:14 smithi112 ceph-mon[32769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T20:27:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:15 smithi130 ceph-mon[32704]: pgmap v94: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:15.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:15 smithi130 ceph-mon[32704]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:27:15.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:15 smithi130 ceph-mon[32704]: Cluster is now healthy 2024-04-26T20:27:15.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:15 smithi130 ceph-mon[32704]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T20:27:15.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:15 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:27:15.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:15 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:27:15.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:15 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:15 smithi102 ceph-mon[28094]: pgmap v94: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:15.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:15 smithi102 ceph-mon[28094]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:27:15.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:15 smithi102 ceph-mon[28094]: Cluster is now healthy 2024-04-26T20:27:15.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:15 smithi102 ceph-mon[28094]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T20:27:15.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:27:15.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:27:15.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:16.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:15 smithi112 ceph-mon[32769]: pgmap v94: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:16.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:15 smithi112 ceph-mon[32769]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:27:16.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:15 smithi112 ceph-mon[32769]: Cluster is now healthy 2024-04-26T20:27:16.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:15 smithi112 ceph-mon[32769]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T20:27:16.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:15 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:27:16.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:15 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:27:16.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:15 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:16.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:16 smithi130 ceph-mon[32704]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:27:16.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:16 smithi102 ceph-mon[28094]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:27:17.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:16 smithi112 ceph-mon[32769]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:27:17.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:17 smithi102 ceph-mon[28094]: pgmap v95: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:17.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:17 smithi130 ceph-mon[32704]: pgmap v95: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:17.927 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:17 smithi112 ceph-mon[32769]: pgmap v95: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:19.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:19 smithi102 ceph-mon[28094]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:19.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:19 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/807916182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6d4daaf-ac9a-4897-a74a-9327aef16d22"}]: dispatch 2024-04-26T20:27:19.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:19 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6d4daaf-ac9a-4897-a74a-9327aef16d22"}]: dispatch 2024-04-26T20:27:19.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:19 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6d4daaf-ac9a-4897-a74a-9327aef16d22"}]': finished 2024-04-26T20:27:19.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:19 smithi102 ceph-mon[28094]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T20:27:19.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:19 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:19.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:19 smithi130 ceph-mon[32704]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:19.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:19 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/807916182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6d4daaf-ac9a-4897-a74a-9327aef16d22"}]: dispatch 2024-04-26T20:27:19.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:19 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6d4daaf-ac9a-4897-a74a-9327aef16d22"}]: dispatch 2024-04-26T20:27:19.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:19 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6d4daaf-ac9a-4897-a74a-9327aef16d22"}]': finished 2024-04-26T20:27:19.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:19 smithi130 ceph-mon[32704]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T20:27:19.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:19 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:20.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:19 smithi112 ceph-mon[32769]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:20.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:19 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/807916182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6d4daaf-ac9a-4897-a74a-9327aef16d22"}]: dispatch 2024-04-26T20:27:20.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:19 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6d4daaf-ac9a-4897-a74a-9327aef16d22"}]: dispatch 2024-04-26T20:27:20.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:19 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6d4daaf-ac9a-4897-a74a-9327aef16d22"}]': finished 2024-04-26T20:27:20.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:19 smithi112 ceph-mon[32769]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T20:27:20.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:19 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:20.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:20 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3666616705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:27:21.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:20 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3666616705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:27:21.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:20 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3666616705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:27:21.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:21 smithi102 ceph-mon[28094]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:22.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:21 smithi112 ceph-mon[32769]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:22.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:21 smithi130 ceph-mon[32704]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:22 smithi102 ceph-mon[28094]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:22.904 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:22 smithi130 ceph-mon[32704]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:23.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:22 smithi112 ceph-mon[32769]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:25.738 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:25 smithi102 ceph-mon[28094]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:25.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:25 smithi112 ceph-mon[32769]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:25.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:25 smithi130 ceph-mon[32704]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:27.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:27 smithi112 ceph-mon[32769]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:27.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:27 smithi102 ceph-mon[28094]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:27.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:27 smithi130 ceph-mon[32704]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:28.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:28 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T20:27:28.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:28 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:28.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T20:27:28.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:28.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T20:27:28.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:29.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:29 smithi102 ceph-mon[28094]: Deploying daemon osd.3 on smithi102 2024-04-26T20:27:29.768 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:29 smithi102 ceph-mon[28094]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:29.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:29 smithi112 ceph-mon[32769]: Deploying daemon osd.3 on smithi102 2024-04-26T20:27:29.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:29 smithi112 ceph-mon[32769]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:29.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:29 smithi130 ceph-mon[32704]: Deploying daemon osd.3 on smithi102 2024-04-26T20:27:29.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:29 smithi130 ceph-mon[32704]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:31.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:31 smithi112 ceph-mon[32769]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:31.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:31 smithi130 ceph-mon[32704]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:31.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:31 smithi102 ceph-mon[28094]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:33.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:33 smithi102 ceph-mon[28094]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:33.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:33 smithi112 ceph-mon[32769]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:33 smithi130 ceph-mon[32704]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:35.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:35 smithi102 ceph-mon[28094]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:35.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:35 smithi112 ceph-mon[32769]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:35.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:35 smithi130 ceph-mon[32704]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:37.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:37 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:37.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:37 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:37.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:37 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:37.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:37 smithi102 ceph-mon[28094]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:37.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:37 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:37.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:37 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:37.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:37 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:37.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:37 smithi112 ceph-mon[32769]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:37.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:37 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:37.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:37 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:37.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:37 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:37.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:37 smithi130 ceph-mon[32704]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:39.717 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:39 smithi102 ceph-mon[28094]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:39.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:39 smithi112 ceph-mon[32769]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:39.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:39 smithi130 ceph-mon[32704]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:40.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.504+0000 7f9440c00700 1 -- 172.21.15.102:0/2938070937 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9444063860 con 0x7f94280a3c40 2024-04-26T20:27:40.506 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 3 on host 'smithi102' 2024-04-26T20:27:40.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.507+0000 7f9449800700 1 -- 172.21.15.102:0/2938070937 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94280a3c40 msgr2=0x7f94280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.507+0000 7f9449800700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94280a3c40 0x7f94280a6100 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9430000c00 tx=0x7f9430005fb0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.508+0000 7f9449800700 1 -- 172.21.15.102:0/2938070937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94441099b0 msgr2=0x7f944419fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:40.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.508+0000 7f9449800700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94441099b0 0x7f944419fab0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f943400ea80 tx=0x7f943400ed90 comp rx=0 tx=0).stop 2024-04-26T20:27:40.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.508+0000 7f9449800700 1 -- 172.21.15.102:0/2938070937 shutdown_connections 2024-04-26T20:27:40.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.508+0000 7f9449800700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94280a3c40 0x7f94280a6100 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.508+0000 7f9449800700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f944410b550 0x7f94441a7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.508+0000 7f9449800700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94441099b0 0x7f944419fab0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.508+0000 7f9449800700 1 --2- 172.21.15.102:0/2938070937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f944410abb0 0x7f944419fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:40.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.508+0000 7f9449800700 1 -- 172.21.15.102:0/2938070937 >> 172.21.15.102:0/2938070937 conn(0x7f9444078980 msgr2=0x7f944410deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:40.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.508+0000 7f9449800700 1 -- 172.21.15.102:0/2938070937 shutdown_connections 2024-04-26T20:27:40.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:27:40.508+0000 7f9449800700 1 -- 172.21.15.102:0/2938070937 wait complete. 2024-04-26T20:27:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:40 smithi112 ceph-mon[32769]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T20:27:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:40 smithi112 ceph-mon[32769]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:41.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:41.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:41.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:40 smithi130 ceph-mon[32704]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T20:27:41.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:40 smithi130 ceph-mon[32704]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:41.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:41.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:41.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:41.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:40 smithi102 ceph-mon[28094]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T20:27:41.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:40 smithi102 ceph-mon[28094]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:41.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:41.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:27:41.304 DEBUG:teuthology.orchestra.run.smithi102:osd.3> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T20:27:41.309 INFO:tasks.cephadm:Deploying osd.4 on smithi112 with /dev/vg_nvme/lv_3... 2024-04-26T20:27:41.309 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T20:27:41.552 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:27:41.834 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:41 smithi102 ceph-mon[28094]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T20:27:41.834 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:41 smithi102 ceph-mon[28094]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T20:27:41.834 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:41 smithi102 ceph-mon[28094]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:27:41.834 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:41 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:41.834 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:27:41 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3[49465]: 2024-04-26T20:27:41.744+0000 7f8bef400700 -1 osd.3 0 waiting for initial osdmap 2024-04-26T20:27:41.835 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:27:41 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3[49465]: 2024-04-26T20:27:41.757+0000 7f8bea200700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:27:42.023 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:41 smithi130 ceph-mon[32704]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T20:27:42.023 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:41 smithi130 ceph-mon[32704]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T20:27:42.023 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:41 smithi130 ceph-mon[32704]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:27:42.023 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:41 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:42.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:41 smithi112 ceph-mon[32769]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T20:27:42.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:41 smithi112 ceph-mon[32769]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T20:27:42.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:41 smithi112 ceph-mon[32769]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:27:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:41 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:42 smithi102 ceph-mon[28094]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:27:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:42 smithi102 ceph-mon[28094]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T20:27:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:42 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:42 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:42 smithi102 ceph-mon[28094]: osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] boot 2024-04-26T20:27:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:42 smithi102 ceph-mon[28094]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T20:27:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:42 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:42.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:42 smithi102 ceph-mon[28094]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:42.986 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:42 smithi112 ceph-mon[32769]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:27:42.986 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:42 smithi112 ceph-mon[32769]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T20:27:42.987 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:42 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:42.987 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:42 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:42.987 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:42 smithi112 ceph-mon[32769]: osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] boot 2024-04-26T20:27:42.987 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:42 smithi112 ceph-mon[32769]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T20:27:42.987 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:42 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:42.987 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:42 smithi112 ceph-mon[32769]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:43.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:42 smithi130 ceph-mon[32704]: from='osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:27:43.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:42 smithi130 ceph-mon[32704]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T20:27:43.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:42 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:43.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:42 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:43.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:42 smithi130 ceph-mon[32704]: osd.3 [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] boot 2024-04-26T20:27:43.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:42 smithi130 ceph-mon[32704]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T20:27:43.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:42 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T20:27:43.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:42 smithi130 ceph-mon[32704]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-26T20:27:43.562 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T20:27:43.562 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T20:27:43.562 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-04-26T20:27:43.562 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-04-26T20:27:43.563 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116385 s, 901 MB/s 2024-04-26T20:27:43.563 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-04-26T20:27:44.212 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:44 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:27:44.212 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:44 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:27:44.212 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:44 smithi112 ceph-mon[32769]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T20:27:44.238 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_3 2024-04-26T20:27:44.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:44 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:27:44.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:44 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:27:44.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:44 smithi130 ceph-mon[32704]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T20:27:44.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:44 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:27:44.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:44 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:27:44.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:44 smithi102 ceph-mon[28094]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T20:27:44.475 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:27:45.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:45 smithi102 ceph-mon[28094]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T20:27:45.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:45 smithi102 ceph-mon[28094]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:27:45.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:45 smithi112 ceph-mon[32769]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T20:27:45.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:45 smithi112 ceph-mon[32769]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:27:45.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:45 smithi130 ceph-mon[32704]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T20:27:45.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:45 smithi130 ceph-mon[32704]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:27:46.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.126+0000 7fde3325a700 1 -- 172.21.15.112:0/1980653257 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde2c103480 msgr2=0x7fde2c1038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:46.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.126+0000 7fde3325a700 1 --2- 172.21.15.112:0/1980653257 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde2c103480 0x7fde2c1038a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fde200099c0 tx=0x7fde20009cd0 comp rx=0 tx=0).stop 2024-04-26T20:27:46.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.126+0000 7fde3325a700 1 -- 172.21.15.112:0/1980653257 shutdown_connections 2024-04-26T20:27:46.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.126+0000 7fde3325a700 1 --2- 172.21.15.112:0/1980653257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c105020 0x7fde2c10b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:46.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.126+0000 7fde3325a700 1 --2- 172.21.15.112:0/1980653257 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde2c103480 0x7fde2c1038a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:46.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.126+0000 7fde3325a700 1 --2- 172.21.15.112:0/1980653257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde2c104680 0x7fde2c104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 -- 172.21.15.112:0/1980653257 >> 172.21.15.112:0/1980653257 conn(0x7fde2c0fea20 msgr2=0x7fde2c100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 -- 172.21.15.112:0/1980653257 shutdown_connections 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 -- 172.21.15.112:0/1980653257 wait complete. 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 Processor -- start 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 -- start start 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde2c103480 0x7fde2c1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c104680 0x7fde2c199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde2c105020 0x7fde2c1a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c10eff0 con 0x7fde2c104680 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c10ee70 con 0x7fde2c103480 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.127+0000 7fde3325a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c10f170 con 0x7fde2c105020 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde30ff6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde2c103480 0x7fde2c1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:46.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde30ff6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde2c103480 0x7fde2c1996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:34730/0 (socket says 172.21.15.112:34730) 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde30ff6700 1 -- 172.21.15.112:0/312052975 learned_addr learned my addr 172.21.15.112:0/312052975 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde30ff6700 1 -- 172.21.15.112:0/312052975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde2c105020 msgr2=0x7fde2c1a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde30ff6700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde2c105020 0x7fde2c1a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde30ff6700 1 -- 172.21.15.112:0/312052975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c104680 msgr2=0x7fde2c199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde30ff6700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c104680 0x7fde2c199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde30ff6700 1 -- 172.21.15.112:0/312052975 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2c1a1320 con 0x7fde2c103480 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde30ff6700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde2c103480 0x7fde2c1996c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde20009f50 tx=0x7fde20003a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde29ffb700 1 -- 172.21.15.112:0/312052975 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde20027070 con 0x7fde2c103480 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde29ffb700 1 -- 172.21.15.112:0/312052975 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde20022470 con 0x7fde2c103480 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde3325a700 1 -- 172.21.15.112:0/312052975 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2c1a15b0 con 0x7fde2c103480 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde29ffb700 1 -- 172.21.15.112:0/312052975 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2000f6c0 con 0x7fde2c103480 2024-04-26T20:27:46.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.128+0000 7fde3325a700 1 -- 172.21.15.112:0/312052975 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2c1a1b60 con 0x7fde2c103480 2024-04-26T20:27:46.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.129+0000 7fde3325a700 1 -- 172.21.15.112:0/312052975 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde2c06a110 con 0x7fde2c103480 2024-04-26T20:27:46.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.130+0000 7fde29ffb700 1 -- 172.21.15.112:0/312052975 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fde2002a070 con 0x7fde2c103480 2024-04-26T20:27:46.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.131+0000 7fde29ffb700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fde1c0a3cc0 0x7fde1c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:27:46.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.131+0000 7fde29ffb700 1 -- 172.21.15.112:0/312052975 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7fde200d0190 con 0x7fde2c103480 2024-04-26T20:27:46.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.131+0000 7fde2bfff700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fde1c0a3cc0 0x7fde1c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:27:46.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.133+0000 7fde2bfff700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fde1c0a3cc0 0x7fde1c0a6180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fde180097b0 tx=0x7fde18006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:27:46.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.136+0000 7fde29ffb700 1 -- 172.21.15.112:0/312052975 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fde20013070 con 0x7fde2c103480 2024-04-26T20:27:46.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:46 smithi112 ceph-mon[32769]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T20:27:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:46.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:27:46.340+0000 7fde3325a700 1 -- 172.21.15.112:0/312052975 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fde2c063860 con 0x7fde1c0a3cc0 2024-04-26T20:27:46.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:46 smithi130 ceph-mon[32704]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T20:27:46.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:46.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:46.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:46.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:46.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:46 smithi102 ceph-mon[28094]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T20:27:46.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:46.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:27:46.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:46.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:46.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:27:46.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:27:47.232 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:47 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi102 2024-04-26T20:27:47.232 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:47 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi102 to 8544M 2024-04-26T20:27:47.233 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:47 smithi112 ceph-mon[32769]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T20:27:47.233 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:47 smithi112 ceph-mon[32769]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:27:47.233 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:27:47.233 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:27:47.233 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:47.233 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:47 smithi112 ceph-mon[32769]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:27:47.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:47 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi102 2024-04-26T20:27:47.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:47 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi102 to 8544M 2024-04-26T20:27:47.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:47 smithi130 ceph-mon[32704]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T20:27:47.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:47 smithi130 ceph-mon[32704]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:27:47.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:27:47.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:27:47.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:47.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:47 smithi130 ceph-mon[32704]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:27:47.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:47 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi102 2024-04-26T20:27:47.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:47 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi102 to 8544M 2024-04-26T20:27:47.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:47 smithi102 ceph-mon[28094]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T20:27:47.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:47 smithi102 ceph-mon[28094]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:27:47.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:27:47.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:27:47.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:47.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:47 smithi102 ceph-mon[28094]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:27:49.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:49 smithi112 ceph-mon[32769]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-26T20:27:49.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:49 smithi112 ceph-mon[32769]: from='client.? 172.21.15.112:0/304230465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fe3b1fe-1d6f-4ce9-b162-fca16bacd628"}]: dispatch 2024-04-26T20:27:49.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:49 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fe3b1fe-1d6f-4ce9-b162-fca16bacd628"}]: dispatch 2024-04-26T20:27:49.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:49 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fe3b1fe-1d6f-4ce9-b162-fca16bacd628"}]': finished 2024-04-26T20:27:49.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:49 smithi112 ceph-mon[32769]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T20:27:49.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:49 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:27:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:49 smithi130 ceph-mon[32704]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-26T20:27:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:49 smithi130 ceph-mon[32704]: from='client.? 172.21.15.112:0/304230465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fe3b1fe-1d6f-4ce9-b162-fca16bacd628"}]: dispatch 2024-04-26T20:27:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:49 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fe3b1fe-1d6f-4ce9-b162-fca16bacd628"}]: dispatch 2024-04-26T20:27:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:49 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fe3b1fe-1d6f-4ce9-b162-fca16bacd628"}]': finished 2024-04-26T20:27:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:49 smithi130 ceph-mon[32704]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T20:27:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:49 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:27:49.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:49 smithi102 ceph-mon[28094]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-26T20:27:49.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:49 smithi102 ceph-mon[28094]: from='client.? 172.21.15.112:0/304230465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fe3b1fe-1d6f-4ce9-b162-fca16bacd628"}]: dispatch 2024-04-26T20:27:49.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:49 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fe3b1fe-1d6f-4ce9-b162-fca16bacd628"}]: dispatch 2024-04-26T20:27:49.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:49 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fe3b1fe-1d6f-4ce9-b162-fca16bacd628"}]': finished 2024-04-26T20:27:49.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:49 smithi102 ceph-mon[28094]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T20:27:49.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:49 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:27:50.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:50 smithi112 ceph-mon[32769]: from='client.? 172.21.15.112:0/3557263749' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:27:50.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:50 smithi130 ceph-mon[32704]: from='client.? 172.21.15.112:0/3557263749' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:27:50.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:50 smithi102 ceph-mon[28094]: from='client.? 172.21.15.112:0/3557263749' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:27:51.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:51 smithi112 ceph-mon[32769]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:27:51.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:51 smithi130 ceph-mon[32704]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:27:51.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:51 smithi102 ceph-mon[28094]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:27:53.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:53 smithi130 ceph-mon[32704]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-26T20:27:53.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:53 smithi112 ceph-mon[32769]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-26T20:27:53.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:53 smithi102 ceph-mon[28094]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-26T20:27:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:55 smithi112 ceph-mon[32769]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-26T20:27:55.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:55 smithi130 ceph-mon[32704]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-26T20:27:55.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:55 smithi102 ceph-mon[28094]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-26T20:27:56.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T20:27:56.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:56.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T20:27:56.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:56.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T20:27:56.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:27:57.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:57 smithi102 ceph-mon[28094]: Deploying daemon osd.4 on smithi112 2024-04-26T20:27:57.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:57 smithi102 ceph-mon[28094]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:27:57.679 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:57 smithi112 ceph-mon[32769]: Deploying daemon osd.4 on smithi112 2024-04-26T20:27:57.680 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:57 smithi112 ceph-mon[32769]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:27:57.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:57 smithi130 ceph-mon[32704]: Deploying daemon osd.4 on smithi112 2024-04-26T20:27:57.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:57 smithi130 ceph-mon[32704]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:27:59.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:27:59 smithi112 ceph-mon[32769]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:27:59.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:27:59 smithi130 ceph-mon[32704]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:27:59.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:27:59 smithi102 ceph-mon[28094]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:01.797 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:01 smithi112 ceph-mon[32769]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:01.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:01 smithi130 ceph-mon[32704]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:01.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:01 smithi102 ceph-mon[28094]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:03.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:03 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:03.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:03 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:03 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:03 smithi112 ceph-mon[32769]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:03.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:03 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:03.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:03 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:03.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:03 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:03.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:03 smithi130 ceph-mon[32704]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:03.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:03.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:03.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:03.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:03 smithi102 ceph-mon[28094]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:05.535 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.533+0000 7fde29ffb700 1 -- 172.21.15.112:0/312052975 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fde2c063860 con 0x7fde1c0a3cc0 2024-04-26T20:28:05.535 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 4 on host 'smithi112' 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 -- 172.21.15.112:0/312052975 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fde1c0a3cc0 msgr2=0x7fde1c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fde1c0a3cc0 0x7fde1c0a6180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fde180097b0 tx=0x7fde18006d20 comp rx=0 tx=0).stop 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 -- 172.21.15.112:0/312052975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde2c103480 msgr2=0x7fde2c1996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde2c103480 0x7fde2c1996c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde20009f50 tx=0x7fde20003a10 comp rx=0 tx=0).stop 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 -- 172.21.15.112:0/312052975 shutdown_connections 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fde1c0a3cc0 0x7fde1c0a6180 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde2c105020 0x7fde2c1a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde2c103480 0x7fde2c1996c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 --2- 172.21.15.112:0/312052975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde2c104680 0x7fde2c199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 -- 172.21.15.112:0/312052975 >> 172.21.15.112:0/312052975 conn(0x7fde2c0fea20 msgr2=0x7fde2c107ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:28:05.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 -- 172.21.15.112:0/312052975 shutdown_connections 2024-04-26T20:28:05.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:28:05.536+0000 7fde3325a700 1 -- 172.21.15.112:0/312052975 wait complete. 2024-04-26T20:28:05.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:05 smithi112 ceph-mon[32769]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:05.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:05 smithi130 ceph-mon[32704]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:05.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:05 smithi102 ceph-mon[28094]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:06.550 DEBUG:teuthology.orchestra.run.smithi112:osd.4> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T20:28:06.553 INFO:tasks.cephadm:Deploying osd.5 on smithi130 with /dev/vg_nvme/lv_3... 2024-04-26T20:28:06.553 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T20:28:06.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:06.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:06.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:06 smithi112 ceph-mon[32769]: from='osd.4 [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:28:06.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:06 smithi112 ceph-mon[32769]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:28:06.797 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:28:06.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:06.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:06.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:06 smithi130 ceph-mon[32704]: from='osd.4 [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:28:06.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:06 smithi130 ceph-mon[32704]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:28:06.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:06.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:06.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:06 smithi102 ceph-mon[28094]: from='osd.4 [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:28:06.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:06 smithi102 ceph-mon[28094]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T20:28:07.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:07 smithi130 ceph-mon[32704]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:07.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:07 smithi130 ceph-mon[32704]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T20:28:07.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:07 smithi130 ceph-mon[32704]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T20:28:07.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:07 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:07.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:07 smithi130 ceph-mon[32704]: from='osd.4 [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:28:07.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:07 smithi130 ceph-mon[32704]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:28:07.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:07 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:07.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:07 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:07.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:07 smithi102 ceph-mon[28094]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:07.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:07 smithi102 ceph-mon[28094]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T20:28:07.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:07 smithi102 ceph-mon[28094]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T20:28:07.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:07 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:07.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:07 smithi102 ceph-mon[28094]: from='osd.4 [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:28:07.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:07 smithi102 ceph-mon[28094]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:28:07.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:07 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:07.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:07 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:07 smithi112 ceph-mon[32769]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:07 smithi112 ceph-mon[32769]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T20:28:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:07 smithi112 ceph-mon[32769]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T20:28:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:07 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:07 smithi112 ceph-mon[32769]: from='osd.4 [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:28:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:07 smithi112 ceph-mon[32769]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:28:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:07 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:07 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:08.446 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:28:08 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4[43865]: 2024-04-26T20:28:08.191+0000 7f18aef5f700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T20:28:08.446 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:28:08 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4[43865]: 2024-04-26T20:28:08.196+0000 7f18a6d4a700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:28:08.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:08 smithi102 ceph-mon[28094]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:28:08.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:08 smithi102 ceph-mon[28094]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T20:28:08.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:08 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:08.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:08 smithi102 ceph-mon[28094]: from='osd.4 ' entity='osd.4' 2024-04-26T20:28:08.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:08 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:08.698 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:08 smithi112 ceph-mon[32769]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:28:08.698 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:08 smithi112 ceph-mon[32769]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T20:28:08.698 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:08 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:08.698 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:08 smithi112 ceph-mon[32769]: from='osd.4 ' entity='osd.4' 2024-04-26T20:28:08.698 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:08 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:08.801 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T20:28:08.801 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T20:28:08.801 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-26T20:28:08.801 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-26T20:28:08.801 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126678 s, 828 MB/s 2024-04-26T20:28:08.801 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-26T20:28:08.801 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:08 smithi130 ceph-mon[32704]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:28:08.801 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:08 smithi130 ceph-mon[32704]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T20:28:08.802 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:08 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:08.802 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:08 smithi130 ceph-mon[32704]: from='osd.4 ' entity='osd.4' 2024-04-26T20:28:08.802 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:08 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:09.370 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi130:vg_nvme/lv_3 2024-04-26T20:28:09.608 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:28:09.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:09 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:28:09.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:09 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:28:09.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:09 smithi130 ceph-mon[32704]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:09.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:09 smithi130 ceph-mon[32704]: osd.4 [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] boot 2024-04-26T20:28:09.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:09 smithi130 ceph-mon[32704]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T20:28:09.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:09 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:09.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:09 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:28:09.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:09 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:28:09.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:09 smithi112 ceph-mon[32769]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:09.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:09 smithi112 ceph-mon[32769]: osd.4 [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] boot 2024-04-26T20:28:09.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:09 smithi112 ceph-mon[32769]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T20:28:09.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:09 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:09 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:28:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:09 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:28:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:09 smithi102 ceph-mon[28094]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-26T20:28:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:09 smithi102 ceph-mon[28094]: osd.4 [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] boot 2024-04-26T20:28:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:09 smithi102 ceph-mon[28094]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T20:28:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T20:28:11.224 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.221+0000 7fb593a7e700 1 -- 172.21.15.130:0/2868959440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb58c103f80 msgr2=0x7fb58c106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:28:11.225 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.221+0000 7fb593a7e700 1 --2- 172.21.15.130:0/2868959440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb58c103f80 0x7fb58c106370 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009990 tx=0x7fb57c009ca0 comp rx=0 tx=0).stop 2024-04-26T20:28:11.225 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.221+0000 7fb593a7e700 1 -- 172.21.15.130:0/2868959440 shutdown_connections 2024-04-26T20:28:11.225 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.221+0000 7fb593a7e700 1 --2- 172.21.15.130:0/2868959440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb58c1091e0 0x7fb58c10b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:11.225 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.221+0000 7fb593a7e700 1 --2- 172.21.15.130:0/2868959440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb58c103f80 0x7fb58c106370 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:11.225 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.221+0000 7fb593a7e700 1 --2- 172.21.15.130:0/2868959440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb58c1068b0 0x7fb58c108ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:11.225 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.221+0000 7fb593a7e700 1 -- 172.21.15.130:0/2868959440 >> 172.21.15.130:0/2868959440 conn(0x7fb58c0fd950 msgr2=0x7fb58c0ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:28:11.225 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.221+0000 7fb593a7e700 1 -- 172.21.15.130:0/2868959440 shutdown_connections 2024-04-26T20:28:11.225 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.221+0000 7fb593a7e700 1 -- 172.21.15.130:0/2868959440 wait complete. 2024-04-26T20:28:11.225 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.222+0000 7fb593a7e700 1 Processor -- start 2024-04-26T20:28:11.225 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.222+0000 7fb593a7e700 1 -- start start 2024-04-26T20:28:11.226 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.222+0000 7fb593a7e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb58c103f80 0x7fb58c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:28:11.226 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.222+0000 7fb593a7e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb58c1068b0 0x7fb58c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:28:11.226 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.222+0000 7fb593a7e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb58c1091e0 0x7fb58c1a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:28:11.226 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.222+0000 7fb593a7e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c10f060 con 0x7fb58c1091e0 2024-04-26T20:28:11.226 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.222+0000 7fb593a7e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c10eee0 con 0x7fb58c1068b0 2024-04-26T20:28:11.226 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.222+0000 7fb593a7e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c10f1e0 con 0x7fb58c103f80 2024-04-26T20:28:11.226 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb59181a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb58c103f80 0x7fb58c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:28:11.226 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb59181a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb58c103f80 0x7fb58c1997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:56472/0 (socket says 172.21.15.130:56472) 2024-04-26T20:28:11.226 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb59181a700 1 -- 172.21.15.130:0/1008520338 learned_addr learned my addr 172.21.15.130:0/1008520338 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-26T20:28:11.227 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb591019700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb58c1068b0 0x7fb58c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:28:11.227 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb59181a700 1 -- 172.21.15.130:0/1008520338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb58c1068b0 msgr2=0x7fb58c199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:28:11.227 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb59181a700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb58c1068b0 0x7fb58c199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:11.227 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb59181a700 1 -- 172.21.15.130:0/1008520338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb58c1091e0 msgr2=0x7fb58c1a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:28:11.227 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb59181a700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb58c1091e0 0x7fb58c1a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:11.227 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb59181a700 1 -- 172.21.15.130:0/1008520338 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58c1a1410 con 0x7fb58c103f80 2024-04-26T20:28:11.227 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb59181a700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb58c103f80 0x7fb58c1997b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb57c000c00 tx=0x7fb57c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:28:11.227 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb582ffd700 1 -- 172.21.15.130:0/1008520338 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c027070 con 0x7fb58c103f80 2024-04-26T20:28:11.227 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb582ffd700 1 -- 172.21.15.130:0/1008520338 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb57c022470 con 0x7fb58c103f80 2024-04-26T20:28:11.228 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb582ffd700 1 -- 172.21.15.130:0/1008520338 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c004230 con 0x7fb58c103f80 2024-04-26T20:28:11.228 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb593a7e700 1 -- 172.21.15.130:0/1008520338 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb58c1a16a0 con 0x7fb58c103f80 2024-04-26T20:28:11.228 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.223+0000 7fb593a7e700 1 -- 172.21.15.130:0/1008520338 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb58c1a1bd0 con 0x7fb58c103f80 2024-04-26T20:28:11.229 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.224+0000 7fb593a7e700 1 -- 172.21.15.130:0/1008520338 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb58c06a110 con 0x7fb58c103f80 2024-04-26T20:28:11.229 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.225+0000 7fb582ffd700 1 -- 172.21.15.130:0/1008520338 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fb57c02a070 con 0x7fb58c103f80 2024-04-26T20:28:11.229 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.226+0000 7fb582ffd700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5780a3cc0 0x7fb5780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:28:11.229 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.226+0000 7fb582ffd700 1 -- 172.21.15.130:0/1008520338 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fb57c0d0630 con 0x7fb58c103f80 2024-04-26T20:28:11.229 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.226+0000 7fb591019700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5780a3cc0 0x7fb5780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:28:11.229 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.227+0000 7fb591019700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5780a3cc0 0x7fb5780a6180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb588006fd0 tx=0x7fb588008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:28:11.230 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.230+0000 7fb582ffd700 1 -- 172.21.15.130:0/1008520338 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb57c013070 con 0x7fb58c103f80 2024-04-26T20:28:11.445 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:11.445+0000 7fb593a7e700 1 -- 172.21.15.130:0/1008520338 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb58c063920 con 0x7fb5780a3cc0 2024-04-26T20:28:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi112 2024-04-26T20:28:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi112 to 8544M 2024-04-26T20:28:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:11.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:11.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:11.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T20:28:11.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:11.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:11.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:11.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi112 2024-04-26T20:28:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi112 to 8544M 2024-04-26T20:28:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:11.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:11.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T20:28:11.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:11.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:11.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:11.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi112 2024-04-26T20:28:11.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi112 to 8544M 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:11.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:12 smithi112 ceph-mon[32769]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:28:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:28:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:28:12.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:12.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:12 smithi130 ceph-mon[32704]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:28:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:28:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:28:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:12.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:12 smithi102 ceph-mon[28094]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:28:12.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:28:12.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:28:12.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:13.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:13 smithi130 ceph-mon[32704]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:13.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:13 smithi112 ceph-mon[32769]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:13.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:13 smithi102 ceph-mon[28094]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:14.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:14 smithi112 ceph-mon[32769]: from='client.? 172.21.15.130:0/3798012831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d936c15-c743-4cef-95fc-f394bf75cdf1"}]: dispatch 2024-04-26T20:28:14.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:14 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d936c15-c743-4cef-95fc-f394bf75cdf1"}]: dispatch 2024-04-26T20:28:14.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:14 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d936c15-c743-4cef-95fc-f394bf75cdf1"}]': finished 2024-04-26T20:28:14.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:14 smithi112 ceph-mon[32769]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T20:28:14.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:14.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:14 smithi130 ceph-mon[32704]: from='client.? 172.21.15.130:0/3798012831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d936c15-c743-4cef-95fc-f394bf75cdf1"}]: dispatch 2024-04-26T20:28:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:14 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d936c15-c743-4cef-95fc-f394bf75cdf1"}]: dispatch 2024-04-26T20:28:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:14 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d936c15-c743-4cef-95fc-f394bf75cdf1"}]': finished 2024-04-26T20:28:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:14 smithi130 ceph-mon[32704]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T20:28:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:14.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:14 smithi102 ceph-mon[28094]: from='client.? 172.21.15.130:0/3798012831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d936c15-c743-4cef-95fc-f394bf75cdf1"}]: dispatch 2024-04-26T20:28:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:14 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d936c15-c743-4cef-95fc-f394bf75cdf1"}]: dispatch 2024-04-26T20:28:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:14 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d936c15-c743-4cef-95fc-f394bf75cdf1"}]': finished 2024-04-26T20:28:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:14 smithi102 ceph-mon[28094]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T20:28:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:15.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:15 smithi112 ceph-mon[32769]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:15.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:15 smithi112 ceph-mon[32769]: from='client.? 172.21.15.130:0/3232016445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:28:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:15 smithi130 ceph-mon[32704]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:15 smithi130 ceph-mon[32704]: from='client.? 172.21.15.130:0/3232016445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:28:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:15 smithi102 ceph-mon[28094]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:15 smithi102 ceph-mon[28094]: from='client.? 172.21.15.130:0/3232016445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:28:17.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:17 smithi112 ceph-mon[32769]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:17.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:17 smithi130 ceph-mon[32704]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:17.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:17 smithi102 ceph-mon[28094]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:19.570 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:19 smithi130 ceph-mon[32704]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:19.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:19 smithi112 ceph-mon[32769]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:19.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:19 smithi102 ceph-mon[28094]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:21.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:21 smithi130 ceph-mon[32704]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:21.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:21 smithi112 ceph-mon[32769]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:21.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:21 smithi102 ceph-mon[28094]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:22.776 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T20:28:22.777 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:22.777 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:22 smithi130 ceph-mon[32704]: Deploying daemon osd.5 on smithi130 2024-04-26T20:28:22.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:22 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T20:28:22.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:22 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:22.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:22 smithi112 ceph-mon[32769]: Deploying daemon osd.5 on smithi130 2024-04-26T20:28:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:22 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T20:28:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:22 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:22 smithi102 ceph-mon[28094]: Deploying daemon osd.5 on smithi130 2024-04-26T20:28:23.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:23 smithi112 ceph-mon[32769]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:23.825 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:23 smithi130 ceph-mon[32704]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:23.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:23 smithi102 ceph-mon[28094]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:25.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:25 smithi130 ceph-mon[32704]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:25.737 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:25 smithi102 ceph-mon[28094]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:25.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:25 smithi112 ceph-mon[32769]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:27.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:27 smithi102 ceph-mon[28094]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:27.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:27.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:27.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:27.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:27 smithi112 ceph-mon[32769]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:27.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:27.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:27.778 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:27 smithi130 ceph-mon[32704]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:27.778 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:27.778 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:27.778 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:29.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:29 smithi112 ceph-mon[32769]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:29.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:29 smithi130 ceph-mon[32704]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:29.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:29 smithi102 ceph-mon[28094]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:30.581 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.580+0000 7fb582ffd700 1 -- 172.21.15.130:0/1008520338 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb58c063920 con 0x7fb5780a3cc0 2024-04-26T20:28:30.581 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 5 on host 'smithi130' 2024-04-26T20:28:30.583 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.582+0000 7fb593a7e700 1 -- 172.21.15.130:0/1008520338 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5780a3cc0 msgr2=0x7fb5780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:28:30.583 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.582+0000 7fb593a7e700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5780a3cc0 0x7fb5780a6180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb588006fd0 tx=0x7fb588008040 comp rx=0 tx=0).stop 2024-04-26T20:28:30.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.582+0000 7fb593a7e700 1 -- 172.21.15.130:0/1008520338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb58c103f80 msgr2=0x7fb58c1997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:28:30.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.582+0000 7fb593a7e700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb58c103f80 0x7fb58c1997b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb57c000c00 tx=0x7fb57c0047b0 comp rx=0 tx=0).stop 2024-04-26T20:28:30.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.582+0000 7fb593a7e700 1 -- 172.21.15.130:0/1008520338 shutdown_connections 2024-04-26T20:28:30.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.582+0000 7fb593a7e700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5780a3cc0 0x7fb5780a6180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:30.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.582+0000 7fb593a7e700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb58c1091e0 0x7fb58c1a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:30.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.582+0000 7fb593a7e700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb58c103f80 0x7fb58c1997b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:30.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.582+0000 7fb593a7e700 1 --2- 172.21.15.130:0/1008520338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb58c1068b0 0x7fb58c199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:30.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.582+0000 7fb593a7e700 1 -- 172.21.15.130:0/1008520338 >> 172.21.15.130:0/1008520338 conn(0x7fb58c0fd950 msgr2=0x7fb58c104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:28:30.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.583+0000 7fb593a7e700 1 -- 172.21.15.130:0/1008520338 shutdown_connections 2024-04-26T20:28:30.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:28:30.583+0000 7fb593a7e700 1 -- 172.21.15.130:0/1008520338 wait complete. 2024-04-26T20:28:31.258 DEBUG:teuthology.orchestra.run.smithi130:osd.5> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T20:28:31.261 INFO:tasks.cephadm:Deploying osd.6 on smithi102 with /dev/vg_nvme/lv_2... 2024-04-26T20:28:31.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T20:28:31.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:31 smithi102 ceph-mon[28094]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:31 smithi102 ceph-mon[28094]: from='osd.5 [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:28:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:31 smithi102 ceph-mon[28094]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:28:31.692 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:28:31.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:31 smithi130 ceph-mon[32704]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:31 smithi130 ceph-mon[32704]: from='osd.5 [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:28:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:31 smithi130 ceph-mon[32704]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:28:32.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:31 smithi112 ceph-mon[32769]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:31 smithi112 ceph-mon[32769]: from='osd.5 [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:28:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:31 smithi112 ceph-mon[32769]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T20:28:32.747 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:32 smithi130 ceph-mon[32704]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T20:28:32.747 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:32 smithi130 ceph-mon[32704]: from='osd.5 [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:28:32.747 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:32 smithi130 ceph-mon[32704]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T20:28:32.747 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:32 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:32.747 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:32 smithi130 ceph-mon[32704]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:28:32.748 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:32 smithi130 ceph-mon[32704]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:32.748 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:28:32 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:28:32.654+0000 7f59c45c2700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T20:28:32.748 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:28:32 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:28:32.660+0000 7f59c03b5700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:28:33.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:32 smithi112 ceph-mon[32769]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T20:28:33.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:32 smithi112 ceph-mon[32769]: from='osd.5 [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:28:33.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:32 smithi112 ceph-mon[32769]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T20:28:33.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:32 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:33.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:32 smithi112 ceph-mon[32769]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:28:33.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:32 smithi112 ceph-mon[32769]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:33.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:32 smithi102 ceph-mon[28094]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T20:28:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:32 smithi102 ceph-mon[28094]: from='osd.5 [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:28:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:32 smithi102 ceph-mon[28094]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T20:28:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:32 smithi102 ceph-mon[28094]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:28:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:32 smithi102 ceph-mon[28094]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-26T20:28:33.838 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:33 smithi102 ceph-mon[28094]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:28:33.839 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:33 smithi102 ceph-mon[28094]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T20:28:33.839 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:33.839 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:33 smithi102 ceph-mon[28094]: osd.5 [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] boot 2024-04-26T20:28:33.839 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:33 smithi102 ceph-mon[28094]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T20:28:33.839 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:33 smithi130 ceph-mon[32704]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:28:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:33 smithi130 ceph-mon[32704]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T20:28:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:33 smithi130 ceph-mon[32704]: osd.5 [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] boot 2024-04-26T20:28:33.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:33 smithi130 ceph-mon[32704]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T20:28:33.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:34.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:33 smithi112 ceph-mon[32769]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:28:34.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:33 smithi112 ceph-mon[32769]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T20:28:34.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:34.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:33 smithi112 ceph-mon[32769]: osd.5 [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] boot 2024-04-26T20:28:34.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:33 smithi112 ceph-mon[32769]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T20:28:34.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T20:28:34.692 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T20:28:34.693 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T20:28:34.693 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-04-26T20:28:34.693 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-04-26T20:28:34.693 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0414118 s, 253 MB/s 2024-04-26T20:28:34.693 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-04-26T20:28:35.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:28:35.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:28:35.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi130 2024-04-26T20:28:35.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi130 to 8544M 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:35.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi130 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi130 to 8544M 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:35.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:35.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:35.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:35.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:35.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:35.075 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:28:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:28:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi130 2024-04-26T20:28:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:28:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi130 to 8544M 2024-04-26T20:28:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:35.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:35.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:35.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:28:35.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:35.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:28:35.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:28:35.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_2 2024-04-26T20:28:35.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:35 smithi102 ceph-mon[28094]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T20:28:35.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:35 smithi102 ceph-mon[28094]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T20:28:35.982 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:28:36.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:35 smithi112 ceph-mon[32769]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T20:28:36.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:35 smithi112 ceph-mon[32769]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T20:28:36.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:35 smithi130 ceph-mon[32704]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T20:28:36.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:35 smithi130 ceph-mon[32704]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T20:28:36.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:36 smithi102 ceph-mon[28094]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:37.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:36 smithi112 ceph-mon[32769]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:37.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:36 smithi130 ceph-mon[32704]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:37.762 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:37 smithi102 ceph-mon[28094]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T20:28:37.762 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:37 smithi102 ceph-mon[28094]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T20:28:38.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:37 smithi112 ceph-mon[32769]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T20:28:38.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:37 smithi112 ceph-mon[32769]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T20:28:38.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:37 smithi130 ceph-mon[32704]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T20:28:38.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:37 smithi130 ceph-mon[32704]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T20:28:38.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.291+0000 7ff9cd000700 1 -- 172.21.15.102:0/3373441503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9c8105070 msgr2=0x7ff9c810b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:28:38.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.291+0000 7ff9cd000700 1 --2- 172.21.15.102:0/3373441503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9c8105070 0x7ff9c810b920 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000b0d0 tx=0x7ff9c000b3e0 comp rx=0 tx=0).stop 2024-04-26T20:28:38.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.291+0000 7ff9cd000700 1 -- 172.21.15.102:0/3373441503 shutdown_connections 2024-04-26T20:28:38.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.291+0000 7ff9cd000700 1 --2- 172.21.15.102:0/3373441503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9c8105070 0x7ff9c810b920 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:38.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.291+0000 7ff9cd000700 1 --2- 172.21.15.102:0/3373441503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c81034d0 0x7ff9c81038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:38.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.291+0000 7ff9cd000700 1 --2- 172.21.15.102:0/3373441503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9c81046d0 0x7ff9c8104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:38.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.291+0000 7ff9cd000700 1 -- 172.21.15.102:0/3373441503 >> 172.21.15.102:0/3373441503 conn(0x7ff9c80fea70 msgr2=0x7ff9c8100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:28:38.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.291+0000 7ff9cd000700 1 -- 172.21.15.102:0/3373441503 shutdown_connections 2024-04-26T20:28:38.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.291+0000 7ff9cd000700 1 -- 172.21.15.102:0/3373441503 wait complete. 2024-04-26T20:28:38.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.292+0000 7ff9cd000700 1 Processor -- start 2024-04-26T20:28:38.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9cd000700 1 -- start start 2024-04-26T20:28:38.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9cd000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9c81034d0 0x7ff9c819b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:28:38.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9cd000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c81046d0 0x7ff9c819bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:28:38.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9cd000700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9c8105070 0x7ff9c81a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:28:38.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9cd000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c810efc0 con 0x7ff9c81046d0 2024-04-26T20:28:38.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9cd000700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c810ee40 con 0x7ff9c8105070 2024-04-26T20:28:38.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9cd000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c810f140 con 0x7ff9c81034d0 2024-04-26T20:28:38.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9c5e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9c81034d0 0x7ff9c819b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:28:38.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9c5400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c81046d0 0x7ff9c819bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:28:38.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9c5e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9c81034d0 0x7ff9c819b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:58240/0 (socket says 172.21.15.102:58240) 2024-04-26T20:28:38.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9c5e00700 1 -- 172.21.15.102:0/3202273706 learned_addr learned my addr 172.21.15.102:0/3202273706 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:28:38.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.293+0000 7ff9c6800700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9c8105070 0x7ff9c81a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:28:38.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.294+0000 7ff9c5e00700 1 -- 172.21.15.102:0/3202273706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9c8105070 msgr2=0x7ff9c81a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:28:38.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.294+0000 7ff9c5e00700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9c8105070 0x7ff9c81a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:38.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.294+0000 7ff9c5e00700 1 -- 172.21.15.102:0/3202273706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c81046d0 msgr2=0x7ff9c819bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:28:38.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.294+0000 7ff9c5e00700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c81046d0 0x7ff9c819bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:28:38.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.294+0000 7ff9c5e00700 1 -- 172.21.15.102:0/3202273706 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c81a3410 con 0x7ff9c81034d0 2024-04-26T20:28:38.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.294+0000 7ff9c6800700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9c8105070 0x7ff9c81a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:28:38.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.294+0000 7ff9c5e00700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9c81034d0 0x7ff9c819b7b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff9b800d9b0 tx=0x7ff9b800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:28:38.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.294+0000 7ff9bea00700 1 -- 172.21.15.102:0/3202273706 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8017030 con 0x7ff9c81034d0 2024-04-26T20:28:38.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.294+0000 7ff9cd000700 1 -- 172.21.15.102:0/3202273706 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9c81a3700 con 0x7ff9c81034d0 2024-04-26T20:28:38.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.295+0000 7ff9bea00700 1 -- 172.21.15.102:0/3202273706 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9b80044d0 con 0x7ff9c81034d0 2024-04-26T20:28:38.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.295+0000 7ff9bea00700 1 -- 172.21.15.102:0/3202273706 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b80038e0 con 0x7ff9c81034d0 2024-04-26T20:28:38.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.295+0000 7ff9cd000700 1 -- 172.21.15.102:0/3202273706 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9c81a3c90 con 0x7ff9c81034d0 2024-04-26T20:28:38.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.296+0000 7ff9cd000700 1 -- 172.21.15.102:0/3202273706 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9c806a110 con 0x7ff9c81034d0 2024-04-26T20:28:38.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.298+0000 7ff9bea00700 1 -- 172.21.15.102:0/3202273706 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ff9b801a070 con 0x7ff9c81034d0 2024-04-26T20:28:38.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.299+0000 7ff9bea00700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff9ac0a3c40 0x7ff9ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:28:38.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.299+0000 7ff9bea00700 1 -- 172.21.15.102:0/3202273706 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7ff9b80ceff0 con 0x7ff9c81034d0 2024-04-26T20:28:38.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.299+0000 7ff9c5400700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff9ac0a3c40 0x7ff9ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:28:38.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.300+0000 7ff9c5400700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff9ac0a3c40 0x7ff9ac0a6100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4009c60 tx=0x7ff9b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:28:38.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.302+0000 7ff9bea00700 1 -- 172.21.15.102:0/3202273706 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff9b8009730 con 0x7ff9c81034d0 2024-04-26T20:28:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:28:38.553+0000 7ff9cd000700 1 -- 172.21.15.102:0/3202273706 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff9c81a3f50 con 0x7ff9ac0a3c40 2024-04-26T20:28:38.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:38 smithi102 ceph-mon[28094]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T20:28:38.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:38 smithi102 ceph-mon[28094]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:28:38.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:28:38.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:28:38.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:39.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:38 smithi112 ceph-mon[32769]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T20:28:39.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:38 smithi112 ceph-mon[32769]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:28:39.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:28:39.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:28:39.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:39.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:38 smithi130 ceph-mon[32704]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T20:28:39.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:38 smithi130 ceph-mon[32704]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:28:39.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:28:39.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:28:39.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:41.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:41 smithi112 ceph-mon[32769]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-26T20:28:41.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:41 smithi130 ceph-mon[32704]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-26T20:28:41.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:41 smithi102 ceph-mon[28094]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-26T20:28:43.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:43 smithi112 ceph-mon[32769]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T20:28:43.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:43 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3128666702' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bbd0cd9-f5a4-4453-860d-2884ccfa71f6"}]: dispatch 2024-04-26T20:28:43.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:43 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3128666702' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bbd0cd9-f5a4-4453-860d-2884ccfa71f6"}]': finished 2024-04-26T20:28:43.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:43 smithi112 ceph-mon[32769]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T20:28:43.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:28:43.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:43 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/311223074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:28:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:43 smithi130 ceph-mon[32704]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T20:28:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:43 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3128666702' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bbd0cd9-f5a4-4453-860d-2884ccfa71f6"}]: dispatch 2024-04-26T20:28:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:43 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3128666702' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bbd0cd9-f5a4-4453-860d-2884ccfa71f6"}]': finished 2024-04-26T20:28:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:43 smithi130 ceph-mon[32704]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T20:28:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:28:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:43 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/311223074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:28:43.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:43 smithi102 ceph-mon[28094]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T20:28:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:43 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3128666702' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bbd0cd9-f5a4-4453-860d-2884ccfa71f6"}]: dispatch 2024-04-26T20:28:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:43 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3128666702' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bbd0cd9-f5a4-4453-860d-2884ccfa71f6"}]': finished 2024-04-26T20:28:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:43 smithi102 ceph-mon[28094]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T20:28:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:28:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:43 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/311223074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:28:45.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:45 smithi112 ceph-mon[32769]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T20:28:45.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:45 smithi130 ceph-mon[32704]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T20:28:45.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:45 smithi102 ceph-mon[28094]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T20:28:47.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:47 smithi112 ceph-mon[32769]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T20:28:47.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:47 smithi130 ceph-mon[32704]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T20:28:47.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:47 smithi102 ceph-mon[28094]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T20:28:49.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:49 smithi102 ceph-mon[28094]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:49.773 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:49 smithi112 ceph-mon[32769]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:49 smithi130 ceph-mon[32704]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:51.602 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:51 smithi102 ceph-mon[28094]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:51.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:51 smithi112 ceph-mon[32769]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:51.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:51 smithi130 ceph-mon[32704]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:52.726 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:52 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T20:28:52.726 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:52 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:52.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:52 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T20:28:52.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:52 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:53.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:52 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T20:28:53.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:52 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:28:53.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:53 smithi102 ceph-mon[28094]: Deploying daemon osd.6 on smithi102 2024-04-26T20:28:53.811 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:53 smithi102 ceph-mon[28094]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:53.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:53 smithi130 ceph-mon[32704]: Deploying daemon osd.6 on smithi102 2024-04-26T20:28:53.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:53 smithi130 ceph-mon[32704]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:54.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:53 smithi112 ceph-mon[32769]: Deploying daemon osd.6 on smithi102 2024-04-26T20:28:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:53 smithi112 ceph-mon[32769]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:55.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:55 smithi102 ceph-mon[28094]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:55.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:55 smithi130 ceph-mon[32704]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:56.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:55 smithi112 ceph-mon[32769]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:57.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:57 smithi130 ceph-mon[32704]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:57.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:57 smithi102 ceph-mon[28094]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:58.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:57 smithi112 ceph-mon[32769]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:59.839 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:28:59 smithi102 ceph-mon[28094]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:59.842 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:28:59 smithi112 ceph-mon[32769]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:28:59.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:28:59 smithi130 ceph-mon[32704]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:01.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:01.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:01 smithi130 ceph-mon[32704]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:01.376 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:01.376 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:01.376 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:01.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:01 smithi102 ceph-mon[28094]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:01.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:01 smithi112 ceph-mon[32769]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:03.717 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:03 smithi102 ceph-mon[28094]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:03.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:03 smithi112 ceph-mon[32769]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:03.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:03 smithi130 ceph-mon[32704]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:05.156 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 6 on host 'smithi102' 2024-04-26T20:29:05.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.154+0000 7ff9bea00700 1 -- 172.21.15.102:0/3202273706 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff9c81a3f50 con 0x7ff9ac0a3c40 2024-04-26T20:29:05.156 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:05.156 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:05.156 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:04 smithi102 ceph-mon[28094]: from='osd.6 [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T20:29:05.156 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:04 smithi102 ceph-mon[28094]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T20:29:05.156 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:04 smithi102 ceph-mon[28094]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:05.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 -- 172.21.15.102:0/3202273706 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff9ac0a3c40 msgr2=0x7ff9ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:05.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff9ac0a3c40 0x7ff9ac0a6100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4009c60 tx=0x7ff9b4009260 comp rx=0 tx=0).stop 2024-04-26T20:29:05.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 -- 172.21.15.102:0/3202273706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9c81034d0 msgr2=0x7ff9c819b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:05.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9c81034d0 0x7ff9c819b7b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff9b800d9b0 tx=0x7ff9b800dcc0 comp rx=0 tx=0).stop 2024-04-26T20:29:05.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 -- 172.21.15.102:0/3202273706 shutdown_connections 2024-04-26T20:29:05.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff9ac0a3c40 0x7ff9ac0a6100 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:05.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9c8105070 0x7ff9c81a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:05.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9c81034d0 0x7ff9c819b7b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:05.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 --2- 172.21.15.102:0/3202273706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c81046d0 0x7ff9c819bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:05.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 -- 172.21.15.102:0/3202273706 >> 172.21.15.102:0/3202273706 conn(0x7ff9c80fea70 msgr2=0x7ff9c8075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:29:05.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 -- 172.21.15.102:0/3202273706 shutdown_connections 2024-04-26T20:29:05.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:29:05.158+0000 7ff9cd000700 1 -- 172.21.15.102:0/3202273706 wait complete. 2024-04-26T20:29:05.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:05.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:05.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:04 smithi112 ceph-mon[32769]: from='osd.6 [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T20:29:05.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:04 smithi112 ceph-mon[32769]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T20:29:05.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:04 smithi112 ceph-mon[32769]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:05.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:05.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:05.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:04 smithi130 ceph-mon[32704]: from='osd.6 [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T20:29:05.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:04 smithi130 ceph-mon[32704]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T20:29:05.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:04 smithi130 ceph-mon[32704]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:05.925 DEBUG:teuthology.orchestra.run.smithi102:osd.6> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T20:29:05.930 INFO:tasks.cephadm:Deploying osd.7 on smithi112 with /dev/vg_nvme/lv_2... 2024-04-26T20:29:05.930 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T20:29:06.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:05 smithi102 ceph-mon[28094]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T20:29:06.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:05 smithi102 ceph-mon[28094]: from='osd.6 [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:29:06.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:05 smithi102 ceph-mon[28094]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T20:29:06.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:06.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:05 smithi102 ceph-mon[28094]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:29:06.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:06.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:06.174 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:29:06.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:05 smithi112 ceph-mon[32769]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T20:29:06.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:05 smithi112 ceph-mon[32769]: from='osd.6 [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:29:06.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:05 smithi112 ceph-mon[32769]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T20:29:06.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:06.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:05 smithi112 ceph-mon[32769]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:29:06.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:06.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:06.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:05 smithi130 ceph-mon[32704]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T20:29:06.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:05 smithi130 ceph-mon[32704]: from='osd.6 [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:29:06.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:05 smithi130 ceph-mon[32704]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T20:29:06.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:06.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:05 smithi130 ceph-mon[32704]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:29:06.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:06.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:07.069 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:06 smithi102 ceph-mon[28094]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:29:07.069 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:06 smithi102 ceph-mon[28094]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T20:29:07.069 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:07.069 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:07.069 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:06 smithi102 ceph-mon[28094]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:07.069 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:06 smithi102 ceph-mon[28094]: from='osd.6 ' entity='osd.6' 2024-04-26T20:29:07.069 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:29:06 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:29:06.806+0000 7f54e6a00700 -1 osd.6 0 waiting for initial osdmap 2024-04-26T20:29:07.069 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:29:06 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:29:06.832+0000 7f54dc800700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:29:07.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:06 smithi112 ceph-mon[32769]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:29:07.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:06 smithi112 ceph-mon[32769]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T20:29:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:06 smithi112 ceph-mon[32769]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:06 smithi112 ceph-mon[32769]: from='osd.6 ' entity='osd.6' 2024-04-26T20:29:07.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:06 smithi130 ceph-mon[32704]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:29:07.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:06 smithi130 ceph-mon[32704]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T20:29:07.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:07.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:06 smithi130 ceph-mon[32704]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-26T20:29:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:06 smithi130 ceph-mon[32704]: from='osd.6 ' entity='osd.6' 2024-04-26T20:29:08.084 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:07 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:29:08.084 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:07 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:29:08.084 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:07 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:08.084 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:07 smithi102 ceph-mon[28094]: osd.6 [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] boot 2024-04-26T20:29:08.084 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:07 smithi102 ceph-mon[28094]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T20:29:08.085 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:07 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:08.170 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T20:29:08.171 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T20:29:08.171 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-04-26T20:29:08.171 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-04-26T20:29:08.171 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113362 s, 925 MB/s 2024-04-26T20:29:08.171 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-04-26T20:29:08.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:07 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:29:08.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:07 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:29:08.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:07 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:08.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:07 smithi112 ceph-mon[32769]: osd.6 [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] boot 2024-04-26T20:29:08.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:07 smithi112 ceph-mon[32769]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T20:29:08.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:07 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:08.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:07 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:29:08.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:07 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:29:08.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:07 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:08.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:07 smithi130 ceph-mon[32704]: osd.6 [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] boot 2024-04-26T20:29:08.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:07 smithi130 ceph-mon[32704]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T20:29:08.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:07 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T20:29:08.901 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_2 2024-04-26T20:29:09.147 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:29:09.268 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:08 smithi112 ceph-mon[32769]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:09.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:08 smithi130 ceph-mon[32704]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:09.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:08 smithi102 ceph-mon[28094]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:10.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:09 smithi112 ceph-mon[32769]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T20:29:10.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:09 smithi130 ceph-mon[32704]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T20:29:10.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:09 smithi102 ceph-mon[28094]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T20:29:10.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.906+0000 7fe79e42e700 1 -- 172.21.15.112:0/663451500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe7981046a0 msgr2=0x7fe798104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:10.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.906+0000 7fe79e42e700 1 --2- 172.21.15.112:0/663451500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe7981046a0 0x7fe798104b00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe7800099c0 tx=0x7fe780009cd0 comp rx=0 tx=0).stop 2024-04-26T20:29:10.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.906+0000 7fe79e42e700 1 -- 172.21.15.112:0/663451500 shutdown_connections 2024-04-26T20:29:10.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.906+0000 7fe79e42e700 1 --2- 172.21.15.112:0/663451500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe798105040 0x7fe79810b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:10.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.906+0000 7fe79e42e700 1 --2- 172.21.15.112:0/663451500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7981034a0 0x7fe7981038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:10.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.906+0000 7fe79e42e700 1 --2- 172.21.15.112:0/663451500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe7981046a0 0x7fe798104b00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:10.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.906+0000 7fe79e42e700 1 -- 172.21.15.112:0/663451500 >> 172.21.15.112:0/663451500 conn(0x7fe7980fea40 msgr2=0x7fe798100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:29:10.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.906+0000 7fe79e42e700 1 -- 172.21.15.112:0/663451500 shutdown_connections 2024-04-26T20:29:10.909 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.906+0000 7fe79e42e700 1 -- 172.21.15.112:0/663451500 wait complete. 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.907+0000 7fe79e42e700 1 Processor -- start 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.907+0000 7fe79e42e700 1 -- start start 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.907+0000 7fe79e42e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7981034a0 0x7fe7981996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.907+0000 7fe79e42e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7981046a0 0x7fe798199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.907+0000 7fe79e42e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe798105040 0x7fe7981a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.907+0000 7fe79e42e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe79810f030 con 0x7fe7981046a0 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.907+0000 7fe79e42e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe79810eeb0 con 0x7fe798105040 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.907+0000 7fe79e42e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe79810f1b0 con 0x7fe7981034a0 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.907+0000 7fe79c9cb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe798105040 0x7fe7981a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe79c9cb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe798105040 0x7fe7981a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:38166/0 (socket says 172.21.15.112:38166) 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe79c9cb700 1 -- 172.21.15.112:0/1803464458 learned_addr learned my addr 172.21.15.112:0/1803464458 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe797fff700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7981034a0 0x7fe7981996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:29:10.910 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe7977fe700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7981046a0 0x7fe798199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe79c9cb700 1 -- 172.21.15.112:0/1803464458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7981034a0 msgr2=0x7fe7981996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe79c9cb700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7981034a0 0x7fe7981996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe79c9cb700 1 -- 172.21.15.112:0/1803464458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7981046a0 msgr2=0x7fe798199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe79c9cb700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7981046a0 0x7fe798199c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe79c9cb700 1 -- 172.21.15.112:0/1803464458 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7981a1320 con 0x7fe798105040 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe79c9cb700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe798105040 0x7fe7981a0c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe79000b970 tx=0x7fe79000bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe7957fa700 1 -- 172.21.15.112:0/1803464458 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790019070 con 0x7fe798105040 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe7957fa700 1 -- 172.21.15.112:0/1803464458 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe79000cde0 con 0x7fe798105040 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe79e42e700 1 -- 172.21.15.112:0/1803464458 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7981a1610 con 0x7fe798105040 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe7957fa700 1 -- 172.21.15.112:0/1803464458 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790007370 con 0x7fe798105040 2024-04-26T20:29:10.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.908+0000 7fe79e42e700 1 -- 172.21.15.112:0/1803464458 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7981a1b80 con 0x7fe798105040 2024-04-26T20:29:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.909+0000 7fe79e42e700 1 -- 172.21.15.112:0/1803464458 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe79806a110 con 0x7fe798105040 2024-04-26T20:29:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.910+0000 7fe7957fa700 1 -- 172.21.15.112:0/1803464458 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe79001c070 con 0x7fe798105040 2024-04-26T20:29:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.911+0000 7fe7957fa700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe7840a3c40 0x7fe7840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:29:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.911+0000 7fe7957fa700 1 -- 172.21.15.112:0/1803464458 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fe7900cf2e0 con 0x7fe798105040 2024-04-26T20:29:10.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.912+0000 7fe797fff700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe7840a3c40 0x7fe7840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:29:10.913 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.913+0000 7fe797fff700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe7840a3c40 0x7fe7840a6100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe798104500 tx=0x7fe788009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:29:10.915 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:10.915+0000 7fe7957fa700 1 -- 172.21.15.112:0/1803464458 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe790023070 con 0x7fe798105040 2024-04-26T20:29:11.125 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:11.125+0000 7fe79e42e700 1 -- 172.21.15.112:0/1803464458 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe798063860 con 0x7fe7840a3c40 2024-04-26T20:29:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:11 smithi112 ceph-mon[32769]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:11 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:11 smithi130 ceph-mon[32704]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:11.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:11.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:11.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:11.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:11.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:11.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:11 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:11 smithi102 ceph-mon[28094]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:11.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:11.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:11.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:11.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:11.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:11.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:11.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:11 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:12.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:12 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi102 2024-04-26T20:29:12.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:12 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi102 to 5696M 2024-04-26T20:29:12.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:12.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:12.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:12.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:12.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:12 smithi112 ceph-mon[32769]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:29:12.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:29:12.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:29:12.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:12 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:12.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:12 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi102 2024-04-26T20:29:12.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:12 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi102 to 5696M 2024-04-26T20:29:12.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:12.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:12.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:12.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:12.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:12 smithi130 ceph-mon[32704]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:29:12.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:29:12.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:29:12.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:12 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:12.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:12 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi102 2024-04-26T20:29:12.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:12 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi102 to 5696M 2024-04-26T20:29:12.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:12.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:12.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:12.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:12.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:12 smithi102 ceph-mon[28094]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:29:12.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:29:12.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:29:12.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:12 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:13.105 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:13 smithi112 ceph-mon[32769]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:13.272 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:13 smithi130 ceph-mon[32704]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:13.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:13 smithi102 ceph-mon[28094]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:14.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:14 smithi112 ceph-mon[32769]: from='client.? 172.21.15.112:0/1466441025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ebc098f-21d8-4b10-a1cb-2793d0675c44"}]: dispatch 2024-04-26T20:29:14.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:14 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ebc098f-21d8-4b10-a1cb-2793d0675c44"}]: dispatch 2024-04-26T20:29:14.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:14 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ebc098f-21d8-4b10-a1cb-2793d0675c44"}]': finished 2024-04-26T20:29:14.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:14 smithi112 ceph-mon[32769]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T20:29:14.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:14.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:14 smithi130 ceph-mon[32704]: from='client.? 172.21.15.112:0/1466441025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ebc098f-21d8-4b10-a1cb-2793d0675c44"}]: dispatch 2024-04-26T20:29:14.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:14 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ebc098f-21d8-4b10-a1cb-2793d0675c44"}]: dispatch 2024-04-26T20:29:14.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:14 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ebc098f-21d8-4b10-a1cb-2793d0675c44"}]': finished 2024-04-26T20:29:14.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:14 smithi130 ceph-mon[32704]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T20:29:14.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:14.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:14 smithi102 ceph-mon[28094]: from='client.? 172.21.15.112:0/1466441025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ebc098f-21d8-4b10-a1cb-2793d0675c44"}]: dispatch 2024-04-26T20:29:14.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:14 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ebc098f-21d8-4b10-a1cb-2793d0675c44"}]: dispatch 2024-04-26T20:29:14.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:14 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ebc098f-21d8-4b10-a1cb-2793d0675c44"}]': finished 2024-04-26T20:29:14.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:14 smithi102 ceph-mon[28094]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T20:29:14.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:15.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:15 smithi130 ceph-mon[32704]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:15.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:15 smithi130 ceph-mon[32704]: from='client.? 172.21.15.112:0/3891720938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:29:15.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:15 smithi102 ceph-mon[28094]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:15.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:15 smithi102 ceph-mon[28094]: from='client.? 172.21.15.112:0/3891720938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:29:15.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:15 smithi112 ceph-mon[32769]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:15.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:15 smithi112 ceph-mon[32769]: from='client.? 172.21.15.112:0/3891720938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:29:17.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:17 smithi112 ceph-mon[32769]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:17.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:17 smithi130 ceph-mon[32704]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:17.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:17 smithi102 ceph-mon[28094]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:19.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:19 smithi112 ceph-mon[32769]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:19.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:19 smithi130 ceph-mon[32704]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:19.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:19 smithi102 ceph-mon[28094]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:21.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:21 smithi112 ceph-mon[32769]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:21.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:21 smithi130 ceph-mon[32704]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:21.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:21 smithi102 ceph-mon[28094]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:22.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:22 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T20:29:22.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:22 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:22.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:22 smithi112 ceph-mon[32769]: Deploying daemon osd.7 on smithi112 2024-04-26T20:29:22.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T20:29:22.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:22 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:22.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:22 smithi130 ceph-mon[32704]: Deploying daemon osd.7 on smithi112 2024-04-26T20:29:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:22 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T20:29:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:22 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:22 smithi102 ceph-mon[28094]: Deploying daemon osd.7 on smithi112 2024-04-26T20:29:23.787 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:23 smithi112 ceph-mon[32769]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:23.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:23 smithi130 ceph-mon[32704]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:23.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:23 smithi102 ceph-mon[28094]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:25.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:25 smithi130 ceph-mon[32704]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:25.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:25 smithi102 ceph-mon[28094]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:26.004 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:25 smithi112 ceph-mon[32769]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:27.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:27 smithi130 ceph-mon[32704]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:27.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:27.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:27 smithi102 ceph-mon[28094]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:27.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:27.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:27.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:28.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:27 smithi112 ceph-mon[32769]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:28.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:29.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:29 smithi112 ceph-mon[32769]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:29.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:29 smithi130 ceph-mon[32704]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:29.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:29 smithi102 ceph-mon[28094]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:31.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:31 smithi112 ceph-mon[32769]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:31 smithi112 ceph-mon[32769]: from='osd.7 [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:29:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:31 smithi112 ceph-mon[32769]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:29:31.457 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 7 on host 'smithi112' 2024-04-26T20:29:31.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.456+0000 7fe7957fa700 1 -- 172.21.15.112:0/1803464458 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe798063860 con 0x7fe7840a3c40 2024-04-26T20:29:31.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 -- 172.21.15.112:0/1803464458 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe7840a3c40 msgr2=0x7fe7840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:31.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe7840a3c40 0x7fe7840a6100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe798104500 tx=0x7fe788009190 comp rx=0 tx=0).stop 2024-04-26T20:29:31.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 -- 172.21.15.112:0/1803464458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe798105040 msgr2=0x7fe7981a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:31.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe798105040 0x7fe7981a0c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe79000b970 tx=0x7fe79000bc80 comp rx=0 tx=0).stop 2024-04-26T20:29:31.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 -- 172.21.15.112:0/1803464458 shutdown_connections 2024-04-26T20:29:31.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe7840a3c40 0x7fe7840a6100 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:31.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe798105040 0x7fe7981a0c80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:31.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7981034a0 0x7fe7981996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:31.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 --2- 172.21.15.112:0/1803464458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7981046a0 0x7fe798199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:31.459 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 -- 172.21.15.112:0/1803464458 >> 172.21.15.112:0/1803464458 conn(0x7fe7980fea40 msgr2=0x7fe798107a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:29:31.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 -- 172.21.15.112:0/1803464458 shutdown_connections 2024-04-26T20:29:31.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:29:31.458+0000 7fe79e42e700 1 -- 172.21.15.112:0/1803464458 wait complete. 2024-04-26T20:29:31.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:31.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:31.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:31.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:31.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:31.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:31 smithi130 ceph-mon[32704]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:31.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:31 smithi130 ceph-mon[32704]: from='osd.7 [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:29:31.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:31 smithi130 ceph-mon[32704]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:29:31.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:31 smithi102 ceph-mon[28094]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:31 smithi102 ceph-mon[28094]: from='osd.7 [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:29:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:31 smithi102 ceph-mon[28094]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T20:29:32.146 DEBUG:teuthology.orchestra.run.smithi112:osd.7> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T20:29:32.148 INFO:tasks.cephadm:Deploying osd.8 on smithi130 with /dev/vg_nvme/lv_2... 2024-04-26T20:29:32.148 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T20:29:32.396 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:29:32.441 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:32 smithi112 ceph-mon[32769]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T20:29:32.442 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:32 smithi112 ceph-mon[32769]: from='osd.7 [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:29:32.442 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:32 smithi112 ceph-mon[32769]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T20:29:32.442 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:32 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:32.442 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:32 smithi112 ceph-mon[32769]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:29:32.442 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:32 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:32.442 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:32 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:32.442 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:32 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:32.442 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:29:32 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:29:32.287+0000 7f9945fdb700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T20:29:32.442 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:29:32 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:29:32.295+0000 7f9941dce700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:29:32.518 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:32 smithi130 ceph-mon[32704]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T20:29:32.518 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:32 smithi130 ceph-mon[32704]: from='osd.7 [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:29:32.518 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:32 smithi130 ceph-mon[32704]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T20:29:32.518 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:32 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:32.518 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:32 smithi130 ceph-mon[32704]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:29:32.518 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:32 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:32.518 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:32 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:32.518 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:32 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:32.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:32 smithi102 ceph-mon[28094]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T20:29:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:32 smithi102 ceph-mon[28094]: from='osd.7 [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:29:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:32 smithi102 ceph-mon[28094]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T20:29:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:32 smithi102 ceph-mon[28094]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:29:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:33.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:33 smithi130 ceph-mon[32704]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:29:33.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:33 smithi130 ceph-mon[32704]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T20:29:33.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:33.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:33 smithi130 ceph-mon[32704]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:33.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:33 smithi130 ceph-mon[32704]: osd.7 [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] boot 2024-04-26T20:29:33.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:33 smithi130 ceph-mon[32704]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T20:29:33.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:33.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:33 smithi102 ceph-mon[28094]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:29:33.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:33 smithi102 ceph-mon[28094]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T20:29:33.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:33.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:33 smithi102 ceph-mon[28094]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:33.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:33 smithi102 ceph-mon[28094]: osd.7 [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] boot 2024-04-26T20:29:33.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:33 smithi102 ceph-mon[28094]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T20:29:33.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:33.660 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:33 smithi112 ceph-mon[32769]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:29:33.660 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:33 smithi112 ceph-mon[32769]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T20:29:33.661 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:33.661 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:33 smithi112 ceph-mon[32769]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T20:29:33.661 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:33 smithi112 ceph-mon[32769]: osd.7 [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] boot 2024-04-26T20:29:33.661 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:33 smithi112 ceph-mon[32769]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T20:29:33.661 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T20:29:34.390 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T20:29:34.390 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T20:29:34.390 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-26T20:29:34.390 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-26T20:29:34.390 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111981 s, 936 MB/s 2024-04-26T20:29:34.390 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-26T20:29:34.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:34 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:29:34.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:34 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:29:34.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:34 smithi130 ceph-mon[32704]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T20:29:34.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:34 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:29:34.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:34 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:29:34.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:34 smithi102 ceph-mon[28094]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T20:29:34.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:34 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:29:34.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:34 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:29:34.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:34 smithi112 ceph-mon[32769]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T20:29:35.098 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi130:vg_nvme/lv_2 2024-04-26T20:29:35.339 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:29:35.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:35 smithi112 ceph-mon[32769]: pgmap v195: 1 pgs: 1 remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:35.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:35.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:35.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:35.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:35 smithi112 ceph-mon[32769]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T20:29:35.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:35.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:35.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:35 smithi130 ceph-mon[32704]: pgmap v195: 1 pgs: 1 remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:35.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:35.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:35.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:35.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:35 smithi130 ceph-mon[32704]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T20:29:35.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:35.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:35.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:35 smithi102 ceph-mon[28094]: pgmap v195: 1 pgs: 1 remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:35.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:35.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:35.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:35.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:35 smithi102 ceph-mon[28094]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T20:29:35.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:35.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:29:36.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:36 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi112 2024-04-26T20:29:36.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:36 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi112 to 5696M 2024-04-26T20:29:36.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:36 smithi130 ceph-mon[32704]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-26T20:29:36.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:36.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:36.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:36.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:36.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:36 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi112 2024-04-26T20:29:36.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:36 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi112 to 5696M 2024-04-26T20:29:36.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:36 smithi102 ceph-mon[28094]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-26T20:29:36.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:36.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:36.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:36.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:36.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:36 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi112 2024-04-26T20:29:36.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:36 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi112 to 5696M 2024-04-26T20:29:36.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:36 smithi112 ceph-mon[32769]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-26T20:29:36.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:36 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:36.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:36 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:36.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:36 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:36.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:36 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:36.908 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.906+0000 7f62bdb65700 1 -- 172.21.15.130:0/1247527555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8103580 msgr2=0x7f62b81039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.906+0000 7f62bdb65700 1 --2- 172.21.15.130:0/1247527555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8103580 0x7f62b81039e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009990 tx=0x7f62ac009ca0 comp rx=0 tx=0).stop 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.906+0000 7f62bdb65700 1 -- 172.21.15.130:0/1247527555 shutdown_connections 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.906+0000 7f62bdb65700 1 --2- 172.21.15.130:0/1247527555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62b8103f20 0x7f62b810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.906+0000 7f62bdb65700 1 --2- 172.21.15.130:0/1247527555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b8102380 0x7f62b81027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.906+0000 7f62bdb65700 1 --2- 172.21.15.130:0/1247527555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8103580 0x7f62b81039e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.906+0000 7f62bdb65700 1 -- 172.21.15.130:0/1247527555 >> 172.21.15.130:0/1247527555 conn(0x7f62b80fd920 msgr2=0x7f62b80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.906+0000 7f62bdb65700 1 -- 172.21.15.130:0/1247527555 shutdown_connections 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.906+0000 7f62bdb65700 1 -- 172.21.15.130:0/1247527555 wait complete. 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.906+0000 7f62bdb65700 1 Processor -- start 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62bdb65700 1 -- start start 2024-04-26T20:29:36.909 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62bdb65700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8102380 0x7f62b806c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62bdb65700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b8103580 0x7f62b819e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62bdb65700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62b8103f20 0x7f62b81a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62bdb65700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f62b810de90 con 0x7f62b8103f20 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62bdb65700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62b810dd10 con 0x7f62b8103580 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62bdb65700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f62b810e010 con 0x7f62b8102380 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8102380 0x7f62b806c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8102380 0x7f62b806c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:52744/0 (socket says 172.21.15.130:52744) 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b77fe700 1 -- 172.21.15.130:0/1881849749 learned_addr learned my addr 172.21.15.130:0/1881849749 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b6ffd700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b8103580 0x7f62b819e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b77fe700 1 -- 172.21.15.130:0/1881849749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b8103580 msgr2=0x7f62b819e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b77fe700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b8103580 0x7f62b819e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b77fe700 1 -- 172.21.15.130:0/1881849749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62b8103f20 msgr2=0x7f62b81a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:36.910 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b77fe700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62b8103f20 0x7f62b81a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:36.911 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b77fe700 1 -- 172.21.15.130:0/1881849749 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b81a1270 con 0x7f62b8102380 2024-04-26T20:29:36.911 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b6ffd700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b8103580 0x7f62b819e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:29:36.911 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b77fe700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8102380 0x7f62b806c6c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62a800efc0 tx=0x7f62a800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:29:36.911 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.907+0000 7f62b4ff9700 1 -- 172.21.15.130:0/1881849749 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a800ca70 con 0x7f62b8102380 2024-04-26T20:29:36.911 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.908+0000 7f62b4ff9700 1 -- 172.21.15.130:0/1881849749 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62a800cc10 con 0x7f62b8102380 2024-04-26T20:29:36.911 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.908+0000 7f62bdb65700 1 -- 172.21.15.130:0/1881849749 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b81a1560 con 0x7f62b8102380 2024-04-26T20:29:36.911 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.908+0000 7f62b4ff9700 1 -- 172.21.15.130:0/1881849749 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8004860 con 0x7f62b8102380 2024-04-26T20:29:36.912 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.908+0000 7f62bdb65700 1 -- 172.21.15.130:0/1881849749 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b81a1ad0 con 0x7f62b8102380 2024-04-26T20:29:36.912 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.908+0000 7f62bdb65700 1 -- 172.21.15.130:0/1881849749 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62b806a110 con 0x7f62b8102380 2024-04-26T20:29:36.912 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.910+0000 7f62b4ff9700 1 -- 172.21.15.130:0/1881849749 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f62a8012050 con 0x7f62b8102380 2024-04-26T20:29:36.912 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.910+0000 7f62b4ff9700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f62a00a3c40 0x7f62a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:29:36.912 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.910+0000 7f62b4ff9700 1 -- 172.21.15.130:0/1881849749 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7f62a80cfad0 con 0x7f62b8102380 2024-04-26T20:29:36.912 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.911+0000 7f62b6ffd700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f62a00a3c40 0x7f62a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:29:36.913 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.912+0000 7f62b6ffd700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f62a00a3c40 0x7f62a00a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00b2e0 tx=0x7f62ac01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:29:36.914 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:36.914+0000 7f62b4ff9700 1 -- 172.21.15.130:0/1881849749 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f62a8010040 con 0x7f62b8102380 2024-04-26T20:29:37.121 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:37.120+0000 7f62bdb65700 1 -- 172.21.15.130:0/1881849749 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f62b8063860 con 0x7f62a00a3c40 2024-04-26T20:29:37.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:37 smithi130 ceph-mon[32704]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T20:29:37.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:37 smithi130 ceph-mon[32704]: pgmap v198: 1 pgs: 1 remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:37.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:37 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:29:37.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:37 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:29:37.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:37 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:37.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:37 smithi102 ceph-mon[28094]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T20:29:37.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:37 smithi102 ceph-mon[28094]: pgmap v198: 1 pgs: 1 remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:37.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:37 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:29:37.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:37 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:29:37.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:37 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:37.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:37 smithi112 ceph-mon[32769]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T20:29:37.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:37 smithi112 ceph-mon[32769]: pgmap v198: 1 pgs: 1 remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:37.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:37 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:29:37.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:37 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:29:37.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:37 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:38.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:38 smithi130 ceph-mon[32704]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:29:38.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:38 smithi130 ceph-mon[32704]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T20:29:38.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:38 smithi102 ceph-mon[28094]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:29:38.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:38 smithi102 ceph-mon[28094]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T20:29:38.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:38 smithi112 ceph-mon[32769]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:29:38.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:38 smithi112 ceph-mon[32769]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T20:29:39.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:39 smithi130 ceph-mon[32704]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:39.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:39 smithi102 ceph-mon[28094]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:39.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:39 smithi112 ceph-mon[32769]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:40.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:40 smithi130 ceph-mon[32704]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-26T20:29:40.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:40 smithi130 ceph-mon[32704]: Cluster is now healthy 2024-04-26T20:29:40.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:40 smithi130 ceph-mon[32704]: from='client.? 172.21.15.130:0/4013994403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a61ad14-24c4-4658-938f-93e58b88ad8a"}]: dispatch 2024-04-26T20:29:40.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:40 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a61ad14-24c4-4658-938f-93e58b88ad8a"}]: dispatch 2024-04-26T20:29:40.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:40 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a61ad14-24c4-4658-938f-93e58b88ad8a"}]': finished 2024-04-26T20:29:40.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:40 smithi130 ceph-mon[32704]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T20:29:40.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:40.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:40 smithi102 ceph-mon[28094]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-26T20:29:40.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:40 smithi102 ceph-mon[28094]: Cluster is now healthy 2024-04-26T20:29:40.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:40 smithi102 ceph-mon[28094]: from='client.? 172.21.15.130:0/4013994403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a61ad14-24c4-4658-938f-93e58b88ad8a"}]: dispatch 2024-04-26T20:29:40.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:40 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a61ad14-24c4-4658-938f-93e58b88ad8a"}]: dispatch 2024-04-26T20:29:40.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:40 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a61ad14-24c4-4658-938f-93e58b88ad8a"}]': finished 2024-04-26T20:29:40.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:40 smithi102 ceph-mon[28094]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T20:29:40.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:40.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:40 smithi112 ceph-mon[32769]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-26T20:29:40.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:40 smithi112 ceph-mon[32769]: Cluster is now healthy 2024-04-26T20:29:40.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:40 smithi112 ceph-mon[32769]: from='client.? 172.21.15.130:0/4013994403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a61ad14-24c4-4658-938f-93e58b88ad8a"}]: dispatch 2024-04-26T20:29:40.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:40 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a61ad14-24c4-4658-938f-93e58b88ad8a"}]: dispatch 2024-04-26T20:29:40.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:40 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a61ad14-24c4-4658-938f-93e58b88ad8a"}]': finished 2024-04-26T20:29:40.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:40 smithi112 ceph-mon[32769]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T20:29:40.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:41.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:41 smithi102 ceph-mon[28094]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:41.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:41 smithi102 ceph-mon[28094]: from='client.? 172.21.15.130:0/2539419158' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:29:41.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:41 smithi112 ceph-mon[32769]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:41.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:41 smithi112 ceph-mon[32769]: from='client.? 172.21.15.130:0/2539419158' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:29:41.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:41 smithi130 ceph-mon[32704]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:41.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:41 smithi130 ceph-mon[32704]: from='client.? 172.21.15.130:0/2539419158' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:29:43.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:43 smithi112 ceph-mon[32769]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:43 smithi130 ceph-mon[32704]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:43.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:43 smithi102 ceph-mon[28094]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:45.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:45 smithi112 ceph-mon[32769]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:45.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:45 smithi130 ceph-mon[32704]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:45.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:45 smithi102 ceph-mon[28094]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:47.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:47 smithi112 ceph-mon[32769]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:47.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T20:29:47.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:47.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:47 smithi130 ceph-mon[32704]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:47.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T20:29:47.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:47.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:47 smithi102 ceph-mon[28094]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:47.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T20:29:47.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:48.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:48 smithi130 ceph-mon[32704]: Deploying daemon osd.8 on smithi130 2024-04-26T20:29:48.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:48 smithi102 ceph-mon[28094]: Deploying daemon osd.8 on smithi130 2024-04-26T20:29:49.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:48 smithi112 ceph-mon[32769]: Deploying daemon osd.8 on smithi130 2024-04-26T20:29:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:49 smithi130 ceph-mon[32704]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:49.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:49 smithi102 ceph-mon[28094]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:50.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:49 smithi112 ceph-mon[32769]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:51 smithi130 ceph-mon[32704]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:51.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:51 smithi102 ceph-mon[28094]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:52.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:51 smithi112 ceph-mon[32769]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:53.727 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:53 smithi130 ceph-mon[32704]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:53.727 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:53.727 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:53.727 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:53 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:53.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:53 smithi102 ceph-mon[28094]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:53.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:53.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:53.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:53 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:54.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:53 smithi112 ceph-mon[32769]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:53 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:55.697 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:55 smithi130 ceph-mon[32704]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:55.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:55 smithi102 ceph-mon[28094]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:56.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:55 smithi112 ceph-mon[32769]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:57.006 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.005+0000 7f62b4ff9700 1 -- 172.21.15.130:0/1881849749 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f62b8063860 con 0x7f62a00a3c40 2024-04-26T20:29:57.007 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 8 on host 'smithi130' 2024-04-26T20:29:57.008 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.007+0000 7f62bdb65700 1 -- 172.21.15.130:0/1881849749 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f62a00a3c40 msgr2=0x7f62a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.007+0000 7f62bdb65700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f62a00a3c40 0x7f62a00a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00b2e0 tx=0x7f62ac01a040 comp rx=0 tx=0).stop 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.007+0000 7f62bdb65700 1 -- 172.21.15.130:0/1881849749 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8102380 msgr2=0x7f62b806c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.007+0000 7f62bdb65700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8102380 0x7f62b806c6c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62a800efc0 tx=0x7f62a800c3a0 comp rx=0 tx=0).stop 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.007+0000 7f62bdb65700 1 -- 172.21.15.130:0/1881849749 shutdown_connections 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.007+0000 7f62bdb65700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f62a00a3c40 0x7f62a00a6100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.007+0000 7f62bdb65700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62b8103f20 0x7f62b81a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.007+0000 7f62bdb65700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62b8102380 0x7f62b806c6c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.007+0000 7f62bdb65700 1 --2- 172.21.15.130:0/1881849749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62b8103580 0x7f62b819e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.007+0000 7f62bdb65700 1 -- 172.21.15.130:0/1881849749 >> 172.21.15.130:0/1881849749 conn(0x7f62b80fd920 msgr2=0x7f62b81068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.008+0000 7f62bdb65700 1 -- 172.21.15.130:0/1881849749 shutdown_connections 2024-04-26T20:29:57.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:29:57.008+0000 7f62bdb65700 1 -- 172.21.15.130:0/1881849749 wait complete. 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: from='osd.8 [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:57.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:56 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.350 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.351 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.351 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:57.351 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:57.351 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.351 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:57.351 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: from='osd.8 [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:29:57.351 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:29:57.351 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.351 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:57.351 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:56 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: from='osd.8 [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:29:57.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:56 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:29:57.677 DEBUG:teuthology.orchestra.run.smithi130:osd.8> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T20:29:57.680 INFO:tasks.cephadm:Deploying osd.9 on smithi102 with /dev/vg_nvme/lv_1... 2024-04-26T20:29:57.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T20:29:58.123 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:29:58.213 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:29:58 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:29:58.043+0000 7fb477f48700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T20:29:58.213 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:29:58 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:29:58.052+0000 7fb473d3b700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:29:58.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:58 smithi130 ceph-mon[32704]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T20:29:58.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:58 smithi130 ceph-mon[32704]: from='osd.8 [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:29:58.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:58 smithi130 ceph-mon[32704]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T20:29:58.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:58 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:58.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:58 smithi130 ceph-mon[32704]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:29:58.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:58 smithi102 ceph-mon[28094]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T20:29:58.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:58 smithi102 ceph-mon[28094]: from='osd.8 [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:29:58.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:58 smithi102 ceph-mon[28094]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T20:29:58.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:58 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:58.296 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:58 smithi102 ceph-mon[28094]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:29:58.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:58 smithi112 ceph-mon[32769]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T20:29:58.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:58 smithi112 ceph-mon[32769]: from='osd.8 [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:29:58.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:58 smithi112 ceph-mon[32769]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T20:29:58.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:58 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:58.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:58 smithi112 ceph-mon[32769]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:29:59.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:59 smithi130 ceph-mon[32704]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:29:59.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:59 smithi130 ceph-mon[32704]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T20:29:59.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:59 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:59.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:59 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:59.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:29:59 smithi130 ceph-mon[32704]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:59.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:59 smithi112 ceph-mon[32769]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:29:59.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:59 smithi112 ceph-mon[32769]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T20:29:59.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:59 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:59.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:59 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:59.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:29:59 smithi112 ceph-mon[32769]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:29:59.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:59 smithi102 ceph-mon[28094]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:29:59.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:59 smithi102 ceph-mon[28094]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T20:29:59.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:59 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:59.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:59 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:29:59.308 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:29:59 smithi102 ceph-mon[28094]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T20:30:00.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:00 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:30:00.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:00 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:30:00.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:00 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:30:00.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:00 smithi102 ceph-mon[28094]: osd.8 [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] boot 2024-04-26T20:30:00.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:00 smithi102 ceph-mon[28094]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T20:30:00.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:00 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:30:00.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:00 smithi102 ceph-mon[28094]: overall HEALTH_OK 2024-04-26T20:30:00.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:00 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:30:00.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:00 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:30:00.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:00 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:30:00.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:00 smithi130 ceph-mon[32704]: osd.8 [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] boot 2024-04-26T20:30:00.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:00 smithi130 ceph-mon[32704]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T20:30:00.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:00 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:30:00.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:00 smithi130 ceph-mon[32704]: overall HEALTH_OK 2024-04-26T20:30:00.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:00 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:30:00.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:00 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:30:00.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:00 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:30:00.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:00 smithi112 ceph-mon[32769]: osd.8 [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] boot 2024-04-26T20:30:00.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:00 smithi112 ceph-mon[32769]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T20:30:00.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:00 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T20:30:00.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:00 smithi112 ceph-mon[32769]: overall HEALTH_OK 2024-04-26T20:30:01.050 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T20:30:01.051 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T20:30:01.051 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-04-26T20:30:01.051 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-04-26T20:30:01.051 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0368978 s, 284 MB/s 2024-04-26T20:30:01.051 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-04-26T20:30:01.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T20:30:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T20:30:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:01.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:01.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T20:30:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:01.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_1 2024-04-26T20:30:02.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:02 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi130 2024-04-26T20:30:02.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:02 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi130 to 5696M 2024-04-26T20:30:02.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:02 smithi102 ceph-mon[28094]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T20:30:02.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:02 smithi102 ceph-mon[28094]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T20:30:02.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:02 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi130 2024-04-26T20:30:02.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:02 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi130 to 5696M 2024-04-26T20:30:02.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:02 smithi130 ceph-mon[32704]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T20:30:02.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:02 smithi130 ceph-mon[32704]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T20:30:02.405 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:30:02.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:02 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi130 2024-04-26T20:30:02.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:02 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi130 to 5696M 2024-04-26T20:30:02.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:02 smithi112 ceph-mon[32769]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T20:30:02.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:02 smithi112 ceph-mon[32769]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T20:30:03.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:03 smithi102 ceph-mon[28094]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:03.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:03 smithi102 ceph-mon[28094]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T20:30:03.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:03 smithi130 ceph-mon[32704]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:03.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:03 smithi130 ceph-mon[32704]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T20:30:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:03 smithi112 ceph-mon[32769]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:03 smithi112 ceph-mon[32769]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T20:30:04.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.549+0000 7f64f4400700 1 -- 172.21.15.102:0/3047027668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64ec103580 msgr2=0x7f64ec1039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:30:04.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.549+0000 7f64f4400700 1 --2- 172.21.15.102:0/3047027668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64ec103580 0x7f64ec1039e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f64d80099c0 tx=0x7f64d8009cd0 comp rx=0 tx=0).stop 2024-04-26T20:30:04.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.549+0000 7f64f4400700 1 -- 172.21.15.102:0/3047027668 shutdown_connections 2024-04-26T20:30:04.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.549+0000 7f64f4400700 1 --2- 172.21.15.102:0/3047027668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64ec103f20 0x7f64ec10a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:04.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.549+0000 7f64f4400700 1 --2- 172.21.15.102:0/3047027668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64ec102380 0x7f64ec1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:04.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.549+0000 7f64f4400700 1 --2- 172.21.15.102:0/3047027668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64ec103580 0x7f64ec1039e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.549+0000 7f64f4400700 1 -- 172.21.15.102:0/3047027668 >> 172.21.15.102:0/3047027668 conn(0x7f64ec0fd940 msgr2=0x7f64ec0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.549+0000 7f64f4400700 1 -- 172.21.15.102:0/3047027668 shutdown_connections 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.549+0000 7f64f4400700 1 -- 172.21.15.102:0/3047027668 wait complete. 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.550+0000 7f64f4400700 1 Processor -- start 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.550+0000 7f64f4400700 1 -- start start 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f4400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64ec102380 0x7f64ec199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f4400700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64ec103580 0x7f64ec199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f4400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64ec103f20 0x7f64ec1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f4400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec10de60 con 0x7f64ec102380 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f4400700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec10dce0 con 0x7f64ec103580 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f4400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec10dfe0 con 0x7f64ec103f20 2024-04-26T20:30:04.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f1a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64ec102380 0x7f64ec199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:30:04.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f1a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64ec102380 0x7f64ec199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51688/0 (socket says 172.21.15.102:51688) 2024-04-26T20:30:04.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f1a00700 1 -- 172.21.15.102:0/1704886089 learned_addr learned my addr 172.21.15.102:0/1704886089 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:30:04.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f1000700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64ec103580 0x7f64ec199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:30:04.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.551+0000 7f64f2400700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64ec103f20 0x7f64ec1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:30:04.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64f2400700 1 -- 172.21.15.102:0/1704886089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64ec103580 msgr2=0x7f64ec199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:30:04.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64f2400700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64ec103580 0x7f64ec199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:04.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64f2400700 1 -- 172.21.15.102:0/1704886089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64ec102380 msgr2=0x7f64ec199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:30:04.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64f2400700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64ec102380 0x7f64ec199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:04.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64f2400700 1 -- 172.21.15.102:0/1704886089 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ec1a1260 con 0x7f64ec103f20 2024-04-26T20:30:04.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64f2400700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64ec103f20 0x7f64ec1a0bc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009f60 tx=0x7f64e4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:30:04.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64ea000700 1 -- 172.21.15.102:0/1704886089 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e40092e0 con 0x7f64ec103f20 2024-04-26T20:30:04.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64f4400700 1 -- 172.21.15.102:0/1704886089 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ec1a14f0 con 0x7f64ec103f20 2024-04-26T20:30:04.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64ea000700 1 -- 172.21.15.102:0/1704886089 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64e4004bb0 con 0x7f64ec103f20 2024-04-26T20:30:04.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64ea000700 1 -- 172.21.15.102:0/1704886089 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4004020 con 0x7f64ec103f20 2024-04-26T20:30:04.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.552+0000 7f64f4400700 1 -- 172.21.15.102:0/1704886089 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ec1a1a80 con 0x7f64ec103f20 2024-04-26T20:30:04.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.553+0000 7f64f4400700 1 -- 172.21.15.102:0/1704886089 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64ec06a110 con 0x7f64ec103f20 2024-04-26T20:30:04.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.555+0000 7f64ea000700 1 -- 172.21.15.102:0/1704886089 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f64e401b030 con 0x7f64ec103f20 2024-04-26T20:30:04.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.556+0000 7f64ea000700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f64d40a3cc0 0x7f64d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:30:04.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.557+0000 7f64ea000700 1 -- 172.21.15.102:0/1704886089 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f64e40cf3a0 con 0x7f64ec103f20 2024-04-26T20:30:04.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.557+0000 7f64f1a00700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f64d40a3cc0 0x7f64d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:30:04.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.558+0000 7f64f1a00700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f64d40a3cc0 0x7f64d40a6180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f64e0007cf0 tx=0x7f64e0007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:30:04.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.561+0000 7f64ea000700 1 -- 172.21.15.102:0/1704886089 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f64e4022020 con 0x7f64ec103f20 2024-04-26T20:30:04.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:04.801+0000 7f64f4400700 1 -- 172.21.15.102:0/1704886089 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f64ec0ffb60 con 0x7f64d40a3cc0 2024-04-26T20:30:05.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:05 smithi112 ceph-mon[32769]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-26T20:30:05.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:30:05.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:30:05.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:05.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:05 smithi130 ceph-mon[32704]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-26T20:30:05.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:30:05.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:30:05.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:05.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:05 smithi102 ceph-mon[28094]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-26T20:30:05.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:30:05.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:30:05.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:06.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:06 smithi130 ceph-mon[32704]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:30:06.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:06 smithi102 ceph-mon[28094]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:30:07.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:06 smithi112 ceph-mon[32769]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:30:07.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:07 smithi130 ceph-mon[32704]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:30:07.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:07 smithi102 ceph-mon[28094]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:30:08.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:07 smithi112 ceph-mon[32769]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:30:09.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:09 smithi130 ceph-mon[32704]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-26T20:30:09.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:09 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2032318175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e80aea-2edc-49c1-8ab8-46d301e58694"}]: dispatch 2024-04-26T20:30:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:09 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e80aea-2edc-49c1-8ab8-46d301e58694"}]: dispatch 2024-04-26T20:30:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:09 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39e80aea-2edc-49c1-8ab8-46d301e58694"}]': finished 2024-04-26T20:30:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:09 smithi130 ceph-mon[32704]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T20:30:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:09 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:09.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:09 smithi102 ceph-mon[28094]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-26T20:30:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:09 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2032318175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e80aea-2edc-49c1-8ab8-46d301e58694"}]: dispatch 2024-04-26T20:30:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:09 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e80aea-2edc-49c1-8ab8-46d301e58694"}]: dispatch 2024-04-26T20:30:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:09 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39e80aea-2edc-49c1-8ab8-46d301e58694"}]': finished 2024-04-26T20:30:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:09 smithi102 ceph-mon[28094]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T20:30:09.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:10.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:09 smithi112 ceph-mon[32769]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-26T20:30:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:09 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2032318175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e80aea-2edc-49c1-8ab8-46d301e58694"}]: dispatch 2024-04-26T20:30:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:09 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e80aea-2edc-49c1-8ab8-46d301e58694"}]: dispatch 2024-04-26T20:30:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:09 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39e80aea-2edc-49c1-8ab8-46d301e58694"}]': finished 2024-04-26T20:30:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:09 smithi112 ceph-mon[32769]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T20:30:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:09 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:10.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:10 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1648482231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:30:10.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:10 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1648482231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:30:11.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:10 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1648482231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:30:11.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:11 smithi130 ceph-mon[32704]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:30:11.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:11 smithi102 ceph-mon[28094]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:30:12.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:11 smithi112 ceph-mon[32769]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T20:30:13.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:13 smithi130 ceph-mon[32704]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-26T20:30:13.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:13 smithi102 ceph-mon[28094]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-26T20:30:14.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:13 smithi112 ceph-mon[32769]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-26T20:30:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:15 smithi130 ceph-mon[32704]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:15 smithi102 ceph-mon[28094]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:16.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:15 smithi112 ceph-mon[32769]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:17.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:17 smithi130 ceph-mon[32704]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:17.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:17 smithi102 ceph-mon[28094]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:18.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:17 smithi112 ceph-mon[32769]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:19.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:19 smithi130 ceph-mon[32704]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:19.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:19 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T20:30:19.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:19 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:19.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:19 smithi130 ceph-mon[32704]: Deploying daemon osd.9 on smithi102 2024-04-26T20:30:19.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:19 smithi102 ceph-mon[28094]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:19.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:19 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T20:30:19.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:19 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:19.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:19 smithi102 ceph-mon[28094]: Deploying daemon osd.9 on smithi102 2024-04-26T20:30:20.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:19 smithi112 ceph-mon[32769]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:20.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:19 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T20:30:20.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:19 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:20.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:19 smithi112 ceph-mon[32769]: Deploying daemon osd.9 on smithi102 2024-04-26T20:30:21.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:21 smithi130 ceph-mon[32704]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:21.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:21 smithi102 ceph-mon[28094]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:22.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:21 smithi112 ceph-mon[32769]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:23.772 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:23 smithi102 ceph-mon[28094]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:23.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:23 smithi130 ceph-mon[32704]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:24.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:23 smithi112 ceph-mon[32769]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:25.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:25 smithi102 ceph-mon[28094]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:25.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:25 smithi130 ceph-mon[32704]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:26.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:25 smithi112 ceph-mon[32769]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:27.805 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:27 smithi102 ceph-mon[28094]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:27.805 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:27.805 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:27.805 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:27.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:27 smithi130 ceph-mon[32704]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:27.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:27.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:27.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:27 smithi112 ceph-mon[32769]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:29.749 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:29 smithi102 ceph-mon[28094]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:29.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:29 smithi130 ceph-mon[32704]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:30.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:29 smithi112 ceph-mon[32769]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:31 smithi130 ceph-mon[32704]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:31 smithi130 ceph-mon[32704]: from='osd.9 [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:30:31.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:31 smithi130 ceph-mon[32704]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:30:31.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:31 smithi102 ceph-mon[28094]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:31.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:31.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:31.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:31 smithi102 ceph-mon[28094]: from='osd.9 [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:30:31.875 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:31 smithi102 ceph-mon[28094]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:30:32.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:31 smithi112 ceph-mon[32769]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:31 smithi112 ceph-mon[32769]: from='osd.9 [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:30:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:31 smithi112 ceph-mon[32769]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T20:30:32.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.355+0000 7f64ea000700 1 -- 172.21.15.102:0/1704886089 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f64ec0ffb60 con 0x7f64d40a3cc0 2024-04-26T20:30:32.357 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 9 on host 'smithi102' 2024-04-26T20:30:32.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 -- 172.21.15.102:0/1704886089 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f64d40a3cc0 msgr2=0x7f64d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:30:32.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f64d40a3cc0 0x7f64d40a6180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f64e0007cf0 tx=0x7f64e0007480 comp rx=0 tx=0).stop 2024-04-26T20:30:32.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 -- 172.21.15.102:0/1704886089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64ec103f20 msgr2=0x7f64ec1a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:30:32.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64ec103f20 0x7f64ec1a0bc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009f60 tx=0x7f64e4009f90 comp rx=0 tx=0).stop 2024-04-26T20:30:32.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 -- 172.21.15.102:0/1704886089 shutdown_connections 2024-04-26T20:30:32.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f64d40a3cc0 0x7f64d40a6180 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:32.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64ec103f20 0x7f64ec1a0bc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:32.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64ec102380 0x7f64ec199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:32.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 --2- 172.21.15.102:0/1704886089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64ec103580 0x7f64ec199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:32.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 -- 172.21.15.102:0/1704886089 >> 172.21.15.102:0/1704886089 conn(0x7f64ec0fd940 msgr2=0x7f64ec0ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:30:32.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 -- 172.21.15.102:0/1704886089 shutdown_connections 2024-04-26T20:30:32.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:30:32.359+0000 7f64f4400700 1 -- 172.21.15.102:0/1704886089 wait complete. 2024-04-26T20:30:33.136 DEBUG:teuthology.orchestra.run.smithi102:osd.9> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T20:30:33.141 INFO:tasks.cephadm:Deploying osd.10 on smithi112 with /dev/vg_nvme/lv_1... 2024-04-26T20:30:33.142 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T20:30:33.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: from='osd.9 [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:33.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: from='osd.9 [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:33.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:33.389 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:30:33.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: from='osd.9 [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:33.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:34.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:34 smithi102 ceph-mon[28094]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:30:34.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:34 smithi102 ceph-mon[28094]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T20:30:34.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:34.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:34.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:34 smithi130 ceph-mon[32704]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:30:34.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:34 smithi130 ceph-mon[32704]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T20:30:34.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:34.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:34.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:34 smithi112 ceph-mon[32769]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-26T20:30:34.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:34 smithi112 ceph-mon[32769]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T20:30:34.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:34.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:35.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:35 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:30:35.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:35 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:30:35.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:35 smithi102 ceph-mon[28094]: osd.9 [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] boot 2024-04-26T20:30:35.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:35 smithi102 ceph-mon[28094]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T20:30:35.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:35.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:35 smithi102 ceph-mon[28094]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:35.079 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:35 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:30:35.079 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:35 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:30:35.079 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:35 smithi112 ceph-mon[32769]: osd.9 [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] boot 2024-04-26T20:30:35.079 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:35 smithi112 ceph-mon[32769]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T20:30:35.079 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:35.080 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:35 smithi112 ceph-mon[32769]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:35.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:35 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:30:35.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:35 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:30:35.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:35 smithi130 ceph-mon[32704]: osd.9 [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] boot 2024-04-26T20:30:35.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:35 smithi130 ceph-mon[32704]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T20:30:35.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T20:30:35.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:35 smithi130 ceph-mon[32704]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-26T20:30:35.571 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T20:30:35.571 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T20:30:35.571 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-04-26T20:30:35.571 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-04-26T20:30:35.571 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113999 s, 920 MB/s 2024-04-26T20:30:35.572 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-04-26T20:30:36.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:36 smithi112 ceph-mon[32769]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T20:30:36.329 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_1 2024-04-26T20:30:36.375 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:36 smithi130 ceph-mon[32704]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T20:30:36.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:36 smithi102 ceph-mon[28094]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T20:30:36.563 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T20:30:37.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:37 smithi112 ceph-mon[32769]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:37.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:37 smithi130 ceph-mon[32704]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:37.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:37 smithi102 ceph-mon[28094]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:38.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.269+0000 7fe880f30700 1 -- 172.21.15.112:0/3481610772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe87c1035a0 msgr2=0x7fe87c103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:30:38.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.269+0000 7fe880f30700 1 --2- 172.21.15.112:0/3481610772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe87c1035a0 0x7fe87c103a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe8700099c0 tx=0x7fe870009cd0 comp rx=0 tx=0).stop 2024-04-26T20:30:38.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.269+0000 7fe880f30700 1 -- 172.21.15.112:0/3481610772 shutdown_connections 2024-04-26T20:30:38.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.269+0000 7fe880f30700 1 --2- 172.21.15.112:0/3481610772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe87c103f40 0x7fe87c10a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:38.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.269+0000 7fe880f30700 1 --2- 172.21.15.112:0/3481610772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe87c1023a0 0x7fe87c1027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:38.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.269+0000 7fe880f30700 1 --2- 172.21.15.112:0/3481610772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe87c1035a0 0x7fe87c103a00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:38.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.269+0000 7fe880f30700 1 -- 172.21.15.112:0/3481610772 >> 172.21.15.112:0/3481610772 conn(0x7fe87c0fd980 msgr2=0x7fe87c0ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:30:38.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.269+0000 7fe880f30700 1 -- 172.21.15.112:0/3481610772 shutdown_connections 2024-04-26T20:30:38.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.269+0000 7fe880f30700 1 -- 172.21.15.112:0/3481610772 wait complete. 2024-04-26T20:30:38.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe880f30700 1 Processor -- start 2024-04-26T20:30:38.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe880f30700 1 -- start start 2024-04-26T20:30:38.273 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe880f30700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe87c1023a0 0x7fe87c06c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe880f30700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe87c1035a0 0x7fe87c19e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe880f30700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe87c103f40 0x7fe87c1a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe880f30700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c10df20 con 0x7fe87c103f40 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe880f30700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c10dda0 con 0x7fe87c1035a0 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe880f30700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c10e0a0 con 0x7fe87c1023a0 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe879d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe87c1035a0 0x7fe87c19e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe879d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe87c1035a0 0x7fe87c19e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39924/0 (socket says 172.21.15.112:39924) 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe879d9b700 1 -- 172.21.15.112:0/241242052 learned_addr learned my addr 172.21.15.112:0/241242052 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe87a59c700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe87c1023a0 0x7fe87c06c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe879d9b700 1 -- 172.21.15.112:0/241242052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe87c1023a0 msgr2=0x7fe87c06c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe879d9b700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe87c1023a0 0x7fe87c06c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe879d9b700 1 -- 172.21.15.112:0/241242052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe87c103f40 msgr2=0x7fe87c1a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:30:38.274 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe879d9b700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe87c103f40 0x7fe87c1a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:38.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.270+0000 7fe879d9b700 1 -- 172.21.15.112:0/241242052 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c1a1310 con 0x7fe87c1035a0 2024-04-26T20:30:38.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.271+0000 7fe879d9b700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe87c1035a0 0x7fe87c19e3b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe870009f50 tx=0x7fe8700047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:30:38.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.271+0000 7fe86b7fe700 1 -- 172.21.15.112:0/241242052 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870027070 con 0x7fe87c1035a0 2024-04-26T20:30:38.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.271+0000 7fe880f30700 1 -- 172.21.15.112:0/241242052 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c1a15a0 con 0x7fe87c1035a0 2024-04-26T20:30:38.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.271+0000 7fe86b7fe700 1 -- 172.21.15.112:0/241242052 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe870022470 con 0x7fe87c1035a0 2024-04-26T20:30:38.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.271+0000 7fe86b7fe700 1 -- 172.21.15.112:0/241242052 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87000f6c0 con 0x7fe87c1035a0 2024-04-26T20:30:38.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.271+0000 7fe880f30700 1 -- 172.21.15.112:0/241242052 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c1a1a00 con 0x7fe87c1035a0 2024-04-26T20:30:38.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.272+0000 7fe880f30700 1 -- 172.21.15.112:0/241242052 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87c06a110 con 0x7fe87c1035a0 2024-04-26T20:30:38.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.273+0000 7fe86b7fe700 1 -- 172.21.15.112:0/241242052 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe87002a070 con 0x7fe87c1035a0 2024-04-26T20:30:38.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.273+0000 7fe86b7fe700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe8640a3c40 0x7fe8640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:30:38.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.273+0000 7fe86b7fe700 1 -- 172.21.15.112:0/241242052 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fe8700d0cb0 con 0x7fe87c1035a0 2024-04-26T20:30:38.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.274+0000 7fe87a59c700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe8640a3c40 0x7fe8640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:30:38.276 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.274+0000 7fe87a59c700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe8640a3c40 0x7fe8640a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe87c103400 tx=0x7fe86c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:30:38.279 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.278+0000 7fe86b7fe700 1 -- 172.21.15.112:0/241242052 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe87005f060 con 0x7fe87c1035a0 2024-04-26T20:30:38.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:38.488+0000 7fe880f30700 1 -- 172.21.15.112:0/241242052 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe87c063860 con 0x7fe8640a3c40 2024-04-26T20:30:39.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi102 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi102 to 4272M 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:30:39.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:30:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:30:39.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:39.374 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi102 2024-04-26T20:30:39.374 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.374 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.374 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.374 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.374 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.374 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.375 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi102 to 4272M 2024-04-26T20:30:39.375 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.375 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:39.375 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:39.375 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.375 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:39.375 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:30:39.375 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:30:39.375 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:30:39.375 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:39.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi102 2024-04-26T20:30:39.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:30:39.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi102 to 4272M 2024-04-26T20:30:39.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:39.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:30:39.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:39.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:39.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:30:39.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:30:39.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:30:39.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:41.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:41 smithi130 ceph-mon[32704]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:41.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:41 smithi130 ceph-mon[32704]: from='client.? 172.21.15.112:0/3027399118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd9ac886-cff6-421d-8d22-fb1680e44c02"}]: dispatch 2024-04-26T20:30:41.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:41 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd9ac886-cff6-421d-8d22-fb1680e44c02"}]: dispatch 2024-04-26T20:30:41.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:41 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd9ac886-cff6-421d-8d22-fb1680e44c02"}]': finished 2024-04-26T20:30:41.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:41 smithi130 ceph-mon[32704]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T20:30:41.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:41 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:30:41.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:41 smithi102 ceph-mon[28094]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:41.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:41 smithi102 ceph-mon[28094]: from='client.? 172.21.15.112:0/3027399118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd9ac886-cff6-421d-8d22-fb1680e44c02"}]: dispatch 2024-04-26T20:30:41.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:41 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd9ac886-cff6-421d-8d22-fb1680e44c02"}]: dispatch 2024-04-26T20:30:41.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:41 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd9ac886-cff6-421d-8d22-fb1680e44c02"}]': finished 2024-04-26T20:30:41.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:41 smithi102 ceph-mon[28094]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T20:30:41.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:41 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:30:42.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:41 smithi112 ceph-mon[32769]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:41 smithi112 ceph-mon[32769]: from='client.? 172.21.15.112:0/3027399118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd9ac886-cff6-421d-8d22-fb1680e44c02"}]: dispatch 2024-04-26T20:30:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:41 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd9ac886-cff6-421d-8d22-fb1680e44c02"}]: dispatch 2024-04-26T20:30:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:41 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd9ac886-cff6-421d-8d22-fb1680e44c02"}]': finished 2024-04-26T20:30:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:41 smithi112 ceph-mon[32769]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T20:30:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:41 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:30:42.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:42 smithi130 ceph-mon[32704]: from='client.? 172.21.15.112:0/2939224819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:30:42.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:42 smithi102 ceph-mon[28094]: from='client.? 172.21.15.112:0/2939224819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:30:43.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:42 smithi112 ceph-mon[32769]: from='client.? 172.21.15.112:0/2939224819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:30:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:43 smithi130 ceph-mon[32704]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:43.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:43 smithi102 ceph-mon[28094]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:44.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:43 smithi112 ceph-mon[32769]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:45.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:45 smithi112 ceph-mon[32769]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:45.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:45 smithi130 ceph-mon[32704]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:45.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:45 smithi102 ceph-mon[28094]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:47.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:47 smithi112 ceph-mon[32769]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:47.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:47 smithi130 ceph-mon[32704]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:47.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:47 smithi102 ceph-mon[28094]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:49.757 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:49 smithi112 ceph-mon[32769]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:49.757 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:49 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T20:30:49.757 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:49 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:49 smithi130 ceph-mon[32704]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:49 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T20:30:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:49 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:49.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:49 smithi102 ceph-mon[28094]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:49.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:49 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T20:30:49.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:49 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:30:50.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:50 smithi130 ceph-mon[32704]: Deploying daemon osd.10 on smithi112 2024-04-26T20:30:50.873 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:50 smithi112 ceph-mon[32769]: Deploying daemon osd.10 on smithi112 2024-04-26T20:30:50.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:50 smithi102 ceph-mon[28094]: Deploying daemon osd.10 on smithi112 2024-04-26T20:30:51.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:51 smithi130 ceph-mon[32704]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:51.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:51 smithi102 ceph-mon[28094]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:51.971 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:51 smithi112 ceph-mon[32769]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:53.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:53 smithi130 ceph-mon[32704]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:53.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:53 smithi102 ceph-mon[28094]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:53 smithi112 ceph-mon[32769]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:55.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:55 smithi112 ceph-mon[32769]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:55.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:55.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:55 smithi130 ceph-mon[32704]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:55.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:55.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:55.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:55.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:55 smithi102 ceph-mon[28094]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:55.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:55.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:30:55.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:57.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:57 smithi102 ceph-mon[28094]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:57.742 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:57 smithi112 ceph-mon[32769]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:57.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:57 smithi130 ceph-mon[32704]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:58.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.826+0000 7fe86b7fe700 1 -- 172.21.15.112:0/241242052 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe87c063860 con 0x7fe8640a3c40 2024-04-26T20:30:58.827 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 10 on host 'smithi112' 2024-04-26T20:30:58.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.828+0000 7fe880f30700 1 -- 172.21.15.112:0/241242052 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe8640a3c40 msgr2=0x7fe8640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:30:58.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.828+0000 7fe880f30700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe8640a3c40 0x7fe8640a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe87c103400 tx=0x7fe86c009190 comp rx=0 tx=0).stop 2024-04-26T20:30:58.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.828+0000 7fe880f30700 1 -- 172.21.15.112:0/241242052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe87c1035a0 msgr2=0x7fe87c19e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:30:58.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.828+0000 7fe880f30700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe87c1035a0 0x7fe87c19e3b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe870009f50 tx=0x7fe8700047b0 comp rx=0 tx=0).stop 2024-04-26T20:30:58.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.828+0000 7fe880f30700 1 -- 172.21.15.112:0/241242052 shutdown_connections 2024-04-26T20:30:58.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.828+0000 7fe880f30700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe8640a3c40 0x7fe8640a6100 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:58.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.828+0000 7fe880f30700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe87c103f40 0x7fe87c1a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:58.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.828+0000 7fe880f30700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe87c1023a0 0x7fe87c06c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:58.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.828+0000 7fe880f30700 1 --2- 172.21.15.112:0/241242052 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe87c1035a0 0x7fe87c19e3b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:30:58.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.828+0000 7fe880f30700 1 -- 172.21.15.112:0/241242052 >> 172.21.15.112:0/241242052 conn(0x7fe87c0fd980 msgr2=0x7fe87c106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:30:58.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.829+0000 7fe880f30700 1 -- 172.21.15.112:0/241242052 shutdown_connections 2024-04-26T20:30:58.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-26T20:30:58.829+0000 7fe880f30700 1 -- 172.21.15.112:0/241242052 wait complete. 2024-04-26T20:30:59.490 DEBUG:teuthology.orchestra.run.smithi112:osd.10> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T20:30:59.493 INFO:tasks.cephadm:Deploying osd.11 on smithi130 with /dev/vg_nvme/lv_1... 2024-04-26T20:30:59.493 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T20:30:59.743 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:30:59.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:59 smithi112 ceph-mon[32769]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:59.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:59 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:59.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:59 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:59.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:59 smithi112 ceph-mon[32769]: from='osd.10 [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:30:59.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:30:59 smithi112 ceph-mon[32769]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:30:59.865 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:59 smithi130 ceph-mon[32704]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:59.866 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:59 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:59.866 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:59 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:59.866 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:59 smithi130 ceph-mon[32704]: from='osd.10 [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:30:59.866 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:30:59 smithi130 ceph-mon[32704]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:30:59.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:59 smithi102 ceph-mon[28094]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:30:59.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:59 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:59.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:59 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:30:59.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:59 smithi102 ceph-mon[28094]: from='osd.10 [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:30:59.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:30:59 smithi102 ceph-mon[28094]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T20:31:00.956 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:00 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:00.956 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:00 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:00.956 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:00 smithi130 ceph-mon[32704]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T20:31:00.956 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:00 smithi130 ceph-mon[32704]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T20:31:00.956 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:00 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:00.956 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:00 smithi130 ceph-mon[32704]: from='osd.10 [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:31:00.956 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:00 smithi130 ceph-mon[32704]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:31:00.956 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:00 smithi130 ceph-mon[32704]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:31:01.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:31:00 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:31:00.801+0000 7f1f021f0700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T20:31:01.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:31:00 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:31:00.814+0000 7f1efdfe3700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:31:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:00 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:00 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:00 smithi112 ceph-mon[32769]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T20:31:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:00 smithi112 ceph-mon[32769]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T20:31:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:00 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:00 smithi112 ceph-mon[32769]: from='osd.10 [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:31:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:00 smithi112 ceph-mon[32769]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:31:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:00 smithi112 ceph-mon[32769]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:31:01.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:00 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:01.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:00 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:01.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:00 smithi102 ceph-mon[28094]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T20:31:01.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:00 smithi102 ceph-mon[28094]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T20:31:01.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:00 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:01.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:00 smithi102 ceph-mon[28094]: from='osd.10 [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:31:01.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:00 smithi102 ceph-mon[28094]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-26T20:31:01.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:00 smithi102 ceph-mon[28094]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:31:01.841 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T20:31:01.841 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T20:31:01.841 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-26T20:31:01.841 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-26T20:31:01.841 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112809 s, 930 MB/s 2024-04-26T20:31:01.841 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-26T20:31:02.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:01 smithi112 ceph-mon[32769]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:31:02.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:01 smithi112 ceph-mon[32769]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T20:31:02.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:02.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:01 smithi130 ceph-mon[32704]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:31:02.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:01 smithi130 ceph-mon[32704]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T20:31:02.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:02.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:01 smithi102 ceph-mon[28094]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-26T20:31:02.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:01 smithi102 ceph-mon[28094]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T20:31:02.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:02.557 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi130:vg_nvme/lv_1 2024-04-26T20:31:02.798 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:31:03.056 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:02 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:31:03.056 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:02 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:31:03.056 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:02 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:03.056 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:02 smithi130 ceph-mon[32704]: osd.10 [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] boot 2024-04-26T20:31:03.056 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:02 smithi130 ceph-mon[32704]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T20:31:03.056 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:02 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:03.056 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:02 smithi130 ceph-mon[32704]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T20:31:03.056 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:02 smithi130 ceph-mon[32704]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:31:03.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:02 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:31:03.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:02 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:31:03.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:02 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:03.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:02 smithi102 ceph-mon[28094]: osd.10 [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] boot 2024-04-26T20:31:03.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:02 smithi102 ceph-mon[28094]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T20:31:03.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:02 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:03.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:02 smithi102 ceph-mon[28094]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T20:31:03.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:02 smithi102 ceph-mon[28094]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:31:03.229 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:02 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:31:03.229 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:02 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:31:03.229 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:02 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:03.229 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:02 smithi112 ceph-mon[32769]: osd.10 [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] boot 2024-04-26T20:31:03.229 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:02 smithi112 ceph-mon[32769]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T20:31:03.229 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:02 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T20:31:03.229 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:02 smithi112 ceph-mon[32769]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T20:31:03.229 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:02 smithi112 ceph-mon[32769]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-26T20:31:04.251 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T20:31:04.251 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi112 2024-04-26T20:31:04.251 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.251 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.251 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi112 to 4272M 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:04.252 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T20:31:04.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi112 2024-04-26T20:31:04.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi112 to 4272M 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:04.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.469+0000 7f380351d700 1 -- 172.21.15.130:0/2347615331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37fc103480 msgr2=0x7f37fc1038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:04.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.469+0000 7f380351d700 1 --2- 172.21.15.130:0/2347615331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37fc103480 0x7f37fc1038a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f37f800b0a0 tx=0x7f37f800b3b0 comp rx=0 tx=0).stop 2024-04-26T20:31:04.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.469+0000 7f380351d700 1 -- 172.21.15.130:0/2347615331 shutdown_connections 2024-04-26T20:31:04.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.469+0000 7f380351d700 1 --2- 172.21.15.130:0/2347615331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37fc105020 0x7f37fc10b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:04.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.469+0000 7f380351d700 1 --2- 172.21.15.130:0/2347615331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37fc103480 0x7f37fc1038a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.469+0000 7f380351d700 1 --2- 172.21.15.130:0/2347615331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37fc104680 0x7f37fc104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.469+0000 7f380351d700 1 -- 172.21.15.130:0/2347615331 >> 172.21.15.130:0/2347615331 conn(0x7f37fc0fea20 msgr2=0x7f37fc100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.469+0000 7f380351d700 1 -- 172.21.15.130:0/2347615331 shutdown_connections 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.469+0000 7f380351d700 1 -- 172.21.15.130:0/2347615331 wait complete. 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f380351d700 1 Processor -- start 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f380351d700 1 -- start start 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f380351d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37fc103480 0x7f37fc077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f380351d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37fc104680 0x7f37fc075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f380351d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37fc105020 0x7f37fc075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f380351d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc10efa0 con 0x7f37fc104680 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f380351d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc10ee20 con 0x7f37fc103480 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f380351d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc10f120 con 0x7f37fc105020 2024-04-26T20:31:04.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f3801aba700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37fc105020 0x7f37fc075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f3801aba700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37fc105020 0x7f37fc075d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:56138/0 (socket says 172.21.15.130:56138) 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f3801aba700 1 -- 172.21.15.130:0/603480115 learned_addr learned my addr 172.21.15.130:0/603480115 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f38012b9700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37fc103480 0x7f37fc077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f3801aba700 1 -- 172.21.15.130:0/603480115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37fc103480 msgr2=0x7f37fc077170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f3801aba700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37fc103480 0x7f37fc077170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f3801aba700 1 -- 172.21.15.130:0/603480115 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37fc104680 msgr2=0x7f37fc075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f3801aba700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37fc104680 0x7f37fc075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f3801aba700 1 -- 172.21.15.130:0/603480115 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37fc076620 con 0x7f37fc105020 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f3801aba700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37fc105020 0x7f37fc075d50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f37f400b550 tx=0x7f37f400b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.470+0000 7f37f27fc700 1 -- 172.21.15.130:0/603480115 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4015020 con 0x7f37fc105020 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.471+0000 7f380351d700 1 -- 172.21.15.130:0/603480115 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37fc1a37f0 con 0x7f37fc105020 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.471+0000 7f37f27fc700 1 -- 172.21.15.130:0/603480115 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f4011ce0 con 0x7f37fc105020 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.471+0000 7f37f27fc700 1 -- 172.21.15.130:0/603480115 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4010dd0 con 0x7f37fc105020 2024-04-26T20:31:04.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.471+0000 7f380351d700 1 -- 172.21.15.130:0/603480115 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37fc1a3d80 con 0x7f37fc105020 2024-04-26T20:31:04.475 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.472+0000 7f380351d700 1 -- 172.21.15.130:0/603480115 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37fc06a110 con 0x7f37fc105020 2024-04-26T20:31:04.475 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.473+0000 7f37f27fc700 1 -- 172.21.15.130:0/603480115 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f37f4018020 con 0x7f37fc105020 2024-04-26T20:31:04.475 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.473+0000 7f37f27fc700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f37e80a3c40 0x7f37e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:04.475 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.473+0000 7f37f27fc700 1 -- 172.21.15.130:0/603480115 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f37f40cf570 con 0x7f37fc105020 2024-04-26T20:31:04.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.474+0000 7f38012b9700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f37e80a3c40 0x7f37e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:04.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.475+0000 7f38012b9700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f37e80a3c40 0x7f37e80a6100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009fd0 tx=0x7f37f800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:04.478 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.478+0000 7f37f27fc700 1 -- 172.21.15.130:0/603480115 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f37f401f020 con 0x7f37fc105020 2024-04-26T20:31:04.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi112 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi112 to 4272M 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:04.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:04.688 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:04.687+0000 7f380351d700 1 -- 172.21.15.130:0/603480115 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f37fc063860 con 0x7f37e80a3c40 2024-04-26T20:31:05.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:05 smithi130 ceph-mon[32704]: pgmap v262: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T20:31:05.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:05 smithi130 ceph-mon[32704]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T20:31:05.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:31:05.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:31:05.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:05 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:05.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:05 smithi130 ceph-mon[32704]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T20:31:05.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:05 smithi102 ceph-mon[28094]: pgmap v262: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T20:31:05.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:05 smithi102 ceph-mon[28094]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T20:31:05.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:31:05.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:31:05.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:05 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:05.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:05 smithi102 ceph-mon[28094]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T20:31:06.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:05 smithi112 ceph-mon[32769]: pgmap v262: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T20:31:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:05 smithi112 ceph-mon[32769]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T20:31:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T20:31:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T20:31:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:05 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:05 smithi112 ceph-mon[32769]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T20:31:07.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:06 smithi112 ceph-mon[32769]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:31:07.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:06 smithi112 ceph-mon[32769]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T20:31:07.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:06 smithi130 ceph-mon[32704]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:31:07.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:06 smithi130 ceph-mon[32704]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T20:31:07.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:06 smithi102 ceph-mon[28094]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:31:07.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:06 smithi102 ceph-mon[28094]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T20:31:07.909 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:07 smithi130 ceph-mon[32704]: pgmap v265: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T20:31:07.909 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:07 smithi130 ceph-mon[32704]: from='client.? 172.21.15.130:0/2704860307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc84bfd4-0926-4025-bb47-9a6a1e648b32"}]: dispatch 2024-04-26T20:31:07.909 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:07 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc84bfd4-0926-4025-bb47-9a6a1e648b32"}]: dispatch 2024-04-26T20:31:07.910 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:07 smithi130 ceph-mon[32704]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc84bfd4-0926-4025-bb47-9a6a1e648b32"}]': finished 2024-04-26T20:31:07.910 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:07 smithi130 ceph-mon[32704]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T20:31:07.910 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:07 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:08.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:07 smithi112 ceph-mon[32769]: pgmap v265: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T20:31:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:07 smithi112 ceph-mon[32769]: from='client.? 172.21.15.130:0/2704860307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc84bfd4-0926-4025-bb47-9a6a1e648b32"}]: dispatch 2024-04-26T20:31:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:07 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc84bfd4-0926-4025-bb47-9a6a1e648b32"}]: dispatch 2024-04-26T20:31:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:07 smithi112 ceph-mon[32769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc84bfd4-0926-4025-bb47-9a6a1e648b32"}]': finished 2024-04-26T20:31:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:07 smithi112 ceph-mon[32769]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T20:31:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:07 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:08.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:07 smithi102 ceph-mon[28094]: pgmap v265: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T20:31:08.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:07 smithi102 ceph-mon[28094]: from='client.? 172.21.15.130:0/2704860307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc84bfd4-0926-4025-bb47-9a6a1e648b32"}]: dispatch 2024-04-26T20:31:08.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:07 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc84bfd4-0926-4025-bb47-9a6a1e648b32"}]: dispatch 2024-04-26T20:31:08.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:07 smithi102 ceph-mon[28094]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc84bfd4-0926-4025-bb47-9a6a1e648b32"}]': finished 2024-04-26T20:31:08.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:07 smithi102 ceph-mon[28094]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T20:31:08.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:07 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:09.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:08 smithi112 ceph-mon[32769]: from='client.? 172.21.15.130:0/3702839173' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:31:09.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:08 smithi130 ceph-mon[32704]: from='client.? 172.21.15.130:0/3702839173' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:31:09.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:08 smithi102 ceph-mon[28094]: from='client.? 172.21.15.130:0/3702839173' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T20:31:10.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:09 smithi112 ceph-mon[32769]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-26T20:31:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:09 smithi112 ceph-mon[32769]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T20:31:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:09 smithi112 ceph-mon[32769]: Cluster is now healthy 2024-04-26T20:31:10.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:09 smithi130 ceph-mon[32704]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-26T20:31:10.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:09 smithi130 ceph-mon[32704]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T20:31:10.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:09 smithi130 ceph-mon[32704]: Cluster is now healthy 2024-04-26T20:31:10.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:09 smithi102 ceph-mon[28094]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-26T20:31:10.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:09 smithi102 ceph-mon[28094]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T20:31:10.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:09 smithi102 ceph-mon[28094]: Cluster is now healthy 2024-04-26T20:31:11.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:10 smithi112 ceph-mon[32769]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:31:11.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:10 smithi130 ceph-mon[32704]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:31:11.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:10 smithi102 ceph-mon[28094]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T20:31:13.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:13 smithi130 ceph-mon[32704]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T20:31:13.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:13 smithi102 ceph-mon[28094]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T20:31:14.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:13 smithi112 ceph-mon[32769]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T20:31:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:15 smithi130 ceph-mon[32704]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T20:31:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:15 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T20:31:15.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:15 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:15 smithi102 ceph-mon[28094]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T20:31:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T20:31:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:16.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:15 smithi112 ceph-mon[32769]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T20:31:16.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:15 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T20:31:16.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:15 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:16.861 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:16 smithi130 ceph-mon[32704]: Deploying daemon osd.11 on smithi130 2024-04-26T20:31:16.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:16 smithi102 ceph-mon[28094]: Deploying daemon osd.11 on smithi130 2024-04-26T20:31:17.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:16 smithi112 ceph-mon[32769]: Deploying daemon osd.11 on smithi130 2024-04-26T20:31:17.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:17 smithi130 ceph-mon[32704]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:31:17.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:17 smithi102 ceph-mon[28094]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:31:17.927 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:17 smithi112 ceph-mon[32769]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T20:31:19.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:19 smithi130 ceph-mon[32704]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-26T20:31:19.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:19 smithi102 ceph-mon[28094]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-26T20:31:20.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:19 smithi112 ceph-mon[32769]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-26T20:31:21.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:21 smithi130 ceph-mon[32704]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:21.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:21.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:21.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:21.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:21 smithi102 ceph-mon[28094]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:21.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:21.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:21.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:22.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:21 smithi112 ceph-mon[32769]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:22.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:22.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:22.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:23.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:23 smithi130 ceph-mon[32704]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:23.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:23 smithi102 ceph-mon[28094]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:23.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:23 smithi112 ceph-mon[32769]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:24.928 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.926+0000 7f37f27fc700 1 -- 172.21.15.130:0/603480115 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f37fc063860 con 0x7f37e80a3c40 2024-04-26T20:31:24.928 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 11 on host 'smithi130' 2024-04-26T20:31:24.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 -- 172.21.15.130:0/603480115 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f37e80a3c40 msgr2=0x7f37e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:24.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f37e80a3c40 0x7f37e80a6100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009fd0 tx=0x7f37f800b7b0 comp rx=0 tx=0).stop 2024-04-26T20:31:24.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 -- 172.21.15.130:0/603480115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37fc105020 msgr2=0x7f37fc075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:24.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37fc105020 0x7f37fc075d50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f37f400b550 tx=0x7f37f400b860 comp rx=0 tx=0).stop 2024-04-26T20:31:24.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 -- 172.21.15.130:0/603480115 shutdown_connections 2024-04-26T20:31:24.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f37e80a3c40 0x7f37e80a6100 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:24.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37fc105020 0x7f37fc075d50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:24.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37fc103480 0x7f37fc077170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:24.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 --2- 172.21.15.130:0/603480115 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37fc104680 0x7f37fc075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:24.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 -- 172.21.15.130:0/603480115 >> 172.21.15.130:0/603480115 conn(0x7f37fc0fea20 msgr2=0x7f37fc107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:24.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 -- 172.21.15.130:0/603480115 shutdown_connections 2024-04-26T20:31:24.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:24.929+0000 7f380351d700 1 -- 172.21.15.130:0/603480115 wait complete. 2024-04-26T20:31:25.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: from='osd.11 [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:31:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:31:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:25.617 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.617 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.617 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:25.617 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:25.617 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.617 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:25.618 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: from='osd.11 [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:31:25.618 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:31:25.618 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.618 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.618 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:25.645 DEBUG:teuthology.orchestra.run.smithi130:osd.11> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T20:31:25.648 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T20:31:25.649 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T20:31:25.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:25.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:25.651 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.651 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:25.651 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: from='osd.11 [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:31:25.651 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T20:31:25.651 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.651 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:25.651 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:26.096 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:31:26.502 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:26 smithi102 ceph-mon[28094]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T20:31:26.502 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:26 smithi102 ceph-mon[28094]: from='osd.11 [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:31:26.502 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:26 smithi102 ceph-mon[28094]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T20:31:26.502 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:26.502 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:26 smithi102 ceph-mon[28094]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:31:26.517 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:26 smithi130 ceph-mon[32704]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T20:31:26.517 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:26 smithi130 ceph-mon[32704]: from='osd.11 [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:31:26.517 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:26 smithi130 ceph-mon[32704]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T20:31:26.517 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:26.517 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:26 smithi130 ceph-mon[32704]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:31:26.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:26 smithi112 ceph-mon[32769]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T20:31:26.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:26 smithi112 ceph-mon[32769]: from='osd.11 [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:31:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:26 smithi112 ceph-mon[32769]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T20:31:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:26 smithi112 ceph-mon[32769]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-26T20:31:27.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:31:26 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:31:26.935+0000 7f71bc1ff700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T20:31:27.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:31:26 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:31:26.949+0000 7f71b3fea700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T20:31:27.422 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:27 smithi130 ceph-mon[32704]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:31:27.422 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:27 smithi130 ceph-mon[32704]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T20:31:27.422 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:27.422 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:27.422 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:27 smithi130 ceph-mon[32704]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:27.423 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:27 smithi130 ceph-mon[32704]: from='osd.11 ' entity='osd.11' 2024-04-26T20:31:27.423 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:27.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:27 smithi112 ceph-mon[32769]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:31:27.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:27 smithi112 ceph-mon[32769]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T20:31:27.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:27.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:27.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:27 smithi112 ceph-mon[32769]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:27.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:27 smithi112 ceph-mon[32769]: from='osd.11 ' entity='osd.11' 2024-04-26T20:31:27.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:27.615 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:27 smithi102 ceph-mon[28094]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-26T20:31:27.616 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:27 smithi102 ceph-mon[28094]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T20:31:27.616 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:27.616 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:27.616 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:27 smithi102 ceph-mon[28094]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-04-26T20:31:27.616 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:27 smithi102 ceph-mon[28094]: from='osd.11 ' entity='osd.11' 2024-04-26T20:31:27.616 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:28.360 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:28 smithi130 ceph-mon[32704]: purged_snaps scrub starts 2024-04-26T20:31:28.360 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:28 smithi130 ceph-mon[32704]: purged_snaps scrub ok 2024-04-26T20:31:28.360 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:28 smithi130 ceph-mon[32704]: osd.11 [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] boot 2024-04-26T20:31:28.360 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:28 smithi130 ceph-mon[32704]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T20:31:28.360 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:28.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.435+0000 7f6a6ae00700 1 -- 172.21.15.102:0/940940793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a640773d0 msgr2=0x7f6a64075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:28.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.435+0000 7f6a6ae00700 1 --2- 172.21.15.102:0/940940793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a640773d0 0x7f6a64075830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009990 tx=0x7f6a54009ca0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.436+0000 7f6a6ae00700 1 -- 172.21.15.102:0/940940793 shutdown_connections 2024-04-26T20:31:28.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.436+0000 7f6a6ae00700 1 --2- 172.21.15.102:0/940940793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a6410b390 0x7f6a6410d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.436+0000 7f6a6ae00700 1 --2- 172.21.15.102:0/940940793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a640773d0 0x7f6a64075830 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.436+0000 7f6a6ae00700 1 --2- 172.21.15.102:0/940940793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a64075e00 0x7f6a64076280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.436+0000 7f6a6ae00700 1 -- 172.21.15.102:0/940940793 >> 172.21.15.102:0/940940793 conn(0x7f6a640fea30 msgr2=0x7f6a64100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:28.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.436+0000 7f6a6ae00700 1 -- 172.21.15.102:0/940940793 shutdown_connections 2024-04-26T20:31:28.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.436+0000 7f6a6ae00700 1 -- 172.21.15.102:0/940940793 wait complete. 2024-04-26T20:31:28.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.437+0000 7f6a6ae00700 1 Processor -- start 2024-04-26T20:31:28.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.437+0000 7f6a6ae00700 1 -- start start 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a6ae00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a64075e00 0x7f6a6419b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a6ae00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a640773d0 0x7f6a6419bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a6ae00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a6410b390 0x7f6a641a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a6ae00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6a64111160 con 0x7f6a64075e00 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a6ae00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6a64110fe0 con 0x7f6a6410b390 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a6ae00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6a641112e0 con 0x7f6a640773d0 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a63e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a64075e00 0x7f6a6419b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a63e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a64075e00 0x7f6a6419b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51818/0 (socket says 172.21.15.102:51818) 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a63e00700 1 -- 172.21.15.102:0/608142581 learned_addr learned my addr 172.21.15.102:0/608142581 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a68e00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a6410b390 0x7f6a641a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:28.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.438+0000 7f6a63400700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a640773d0 0x7f6a6419bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a68e00700 1 -- 172.21.15.102:0/608142581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a640773d0 msgr2=0x7f6a6419bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a68e00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a640773d0 0x7f6a6419bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a68e00700 1 -- 172.21.15.102:0/608142581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a64075e00 msgr2=0x7f6a6419b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a68e00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a64075e00 0x7f6a6419b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a68e00700 1 -- 172.21.15.102:0/608142581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a641a33e0 con 0x7f6a6410b390 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a63e00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a64075e00 0x7f6a6419b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a68e00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a6410b390 0x7f6a641a2d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00efc0 tx=0x7f6a5c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a60c00700 1 -- 172.21.15.102:0/608142581 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c0092e0 con 0x7f6a6410b390 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a60c00700 1 -- 172.21.15.102:0/608142581 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a5c004bb0 con 0x7f6a6410b390 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a6ae00700 1 -- 172.21.15.102:0/608142581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a641a3670 con 0x7f6a6410b390 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.439+0000 7f6a60c00700 1 -- 172.21.15.102:0/608142581 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c004020 con 0x7f6a6410b390 2024-04-26T20:31:28.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.440+0000 7f6a6ae00700 1 -- 172.21.15.102:0/608142581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a641a3bb0 con 0x7f6a6410b390 2024-04-26T20:31:28.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.441+0000 7f6a6ae00700 1 -- 172.21.15.102:0/608142581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a6406a110 con 0x7f6a6410b390 2024-04-26T20:31:28.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.442+0000 7f6a60c00700 1 -- 172.21.15.102:0/608142581 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f6a5c01b030 con 0x7f6a6410b390 2024-04-26T20:31:28.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.444+0000 7f6a60c00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a480a3c40 0x7f6a480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:28.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.444+0000 7f6a60c00700 1 -- 172.21.15.102:0/608142581 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f6a5c0cf760 con 0x7f6a6410b390 2024-04-26T20:31:28.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.444+0000 7f6a63e00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a480a3c40 0x7f6a480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:28.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.445+0000 7f6a63e00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a480a3c40 0x7f6a480a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6a54006010 tx=0x7f6a5400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:28.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.448+0000 7f6a60c00700 1 -- 172.21.15.102:0/608142581 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6a5c022020 con 0x7f6a6410b390 2024-04-26T20:31:28.515 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:28 smithi102 ceph-mon[28094]: purged_snaps scrub starts 2024-04-26T20:31:28.515 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:28 smithi102 ceph-mon[28094]: purged_snaps scrub ok 2024-04-26T20:31:28.515 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:28 smithi102 ceph-mon[28094]: osd.11 [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] boot 2024-04-26T20:31:28.515 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:28 smithi102 ceph-mon[28094]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T20:31:28.516 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:28.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:28 smithi112 ceph-mon[32769]: purged_snaps scrub starts 2024-04-26T20:31:28.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:28 smithi112 ceph-mon[32769]: purged_snaps scrub ok 2024-04-26T20:31:28.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:28 smithi112 ceph-mon[32769]: osd.11 [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] boot 2024-04-26T20:31:28.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:28 smithi112 ceph-mon[32769]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T20:31:28.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:28 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T20:31:28.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.682+0000 7f6a6ae00700 1 -- 172.21.15.102:0/608142581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f6a64064030 con 0x7f6a6410b390 2024-04-26T20:31:28.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.683+0000 7f6a60c00700 1 -- 172.21.15.102:0/608142581 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f6a5c010040 con 0x7f6a6410b390 2024-04-26T20:31:28.684 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:28.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 -- 172.21.15.102:0/608142581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a480a3c40 msgr2=0x7f6a480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:28.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a480a3c40 0x7f6a480a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6a54006010 tx=0x7f6a5400b260 comp rx=0 tx=0).stop 2024-04-26T20:31:28.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 -- 172.21.15.102:0/608142581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a6410b390 msgr2=0x7f6a641a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:28.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a6410b390 0x7f6a641a2d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00efc0 tx=0x7f6a5c00c3a0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 -- 172.21.15.102:0/608142581 shutdown_connections 2024-04-26T20:31:28.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a480a3c40 0x7f6a480a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a6410b390 0x7f6a641a2d40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a64075e00 0x7f6a6419b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 --2- 172.21.15.102:0/608142581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a640773d0 0x7f6a6419bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:28.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 -- 172.21.15.102:0/608142581 >> 172.21.15.102:0/608142581 conn(0x7f6a640fea30 msgr2=0x7f6a641072e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:28.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 -- 172.21.15.102:0/608142581 shutdown_connections 2024-04-26T20:31:28.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:28.686+0000 7f6a6ae00700 1 -- 172.21.15.102:0/608142581 wait complete. 2024-04-26T20:31:29.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:29.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/608142581' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T20:31:29.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:29.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:29.400 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/608142581' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T20:31:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:29.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:29.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.566 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1714163487,"num_in_osds":12,"osd_in_since":1714163467,"num_remapped_pgs":0} 2024-04-26T20:31:29.570 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/608142581' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:29.994 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:31:30.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:30 smithi102 ceph-mon[28094]: Detected new or changed devices on smithi130 2024-04-26T20:31:30.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:30 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi130 to 4272M 2024-04-26T20:31:30.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:30 smithi102 ceph-mon[28094]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T20:31:30.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:30 smithi112 ceph-mon[32769]: Detected new or changed devices on smithi130 2024-04-26T20:31:30.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:30 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi130 to 4272M 2024-04-26T20:31:30.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:30 smithi112 ceph-mon[32769]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T20:31:30.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:30 smithi130 ceph-mon[32704]: Detected new or changed devices on smithi130 2024-04-26T20:31:30.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:30 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi130 to 4272M 2024-04-26T20:31:30.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:30 smithi130 ceph-mon[32704]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T20:31:31.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:31 smithi102 ceph-mon[28094]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:31.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:31 smithi130 ceph-mon[32704]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:31.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:31 smithi112 ceph-mon[32769]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.323+0000 7f165f600700 1 -- 172.21.15.102:0/4100369400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1658102320 msgr2=0x7f16581027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.323+0000 7f165f600700 1 --2- 172.21.15.102:0/4100369400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1658102320 0x7f16581027a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f16400098f0 tx=0x7f1640009c00 comp rx=0 tx=0).stop 2024-04-26T20:31:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.323+0000 7f165f600700 1 -- 172.21.15.102:0/4100369400 shutdown_connections 2024-04-26T20:31:32.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.323+0000 7f165f600700 1 --2- 172.21.15.102:0/4100369400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1658102ce0 0x7f165810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:32.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.323+0000 7f165f600700 1 --2- 172.21.15.102:0/4100369400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1658108390 0x7f1658108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:32.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.323+0000 7f165f600700 1 --2- 172.21.15.102:0/4100369400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1658102320 0x7f16581027a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:32.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.324+0000 7f165f600700 1 -- 172.21.15.102:0/4100369400 >> 172.21.15.102:0/4100369400 conn(0x7f16580792d0 msgr2=0x7f16580796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:32.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.324+0000 7f165f600700 1 -- 172.21.15.102:0/4100369400 shutdown_connections 2024-04-26T20:31:32.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.324+0000 7f165f600700 1 -- 172.21.15.102:0/4100369400 wait complete. 2024-04-26T20:31:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.325+0000 7f165f600700 1 Processor -- start 2024-04-26T20:31:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.325+0000 7f165f600700 1 -- start start 2024-04-26T20:31:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165f600700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1658102320 0x7f165819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165f600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1658102ce0 0x7f165819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165f600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1658108390 0x7f16581998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165f600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f16581124e0 con 0x7f1658102ce0 2024-04-26T20:31:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165f600700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1658112360 con 0x7f1658102320 2024-04-26T20:31:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165f600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1658112660 con 0x7f1658108390 2024-04-26T20:31:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f164fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1658102ce0 0x7f165819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165cc00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1658102320 0x7f165819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165cc00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1658102320 0x7f165819f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:56898/0 (socket says 172.21.15.102:56898) 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165cc00700 1 -- 172.21.15.102:0/1204335686 learned_addr learned my addr 172.21.15.102:0/1204335686 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f164fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1658102ce0 0x7f165819fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51840/0 (socket says 172.21.15.102:51840) 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165d600700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1658108390 0x7f16581998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165cc00700 1 -- 172.21.15.102:0/1204335686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1658108390 msgr2=0x7f16581998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165cc00700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1658108390 0x7f16581998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165cc00700 1 -- 172.21.15.102:0/1204335686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1658102ce0 msgr2=0x7f165819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.326+0000 7f165cc00700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1658102ce0 0x7f165819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.327+0000 7f165cc00700 1 -- 172.21.15.102:0/1204335686 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165819a030 con 0x7f1658102320 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.327+0000 7f164fe00700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1658102ce0 0x7f165819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:31:32.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.327+0000 7f165cc00700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1658102320 0x7f165819f710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f164800ec10 tx=0x7f164800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:32.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.327+0000 7f1656000700 1 -- 172.21.15.102:0/1204335686 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164800cc90 con 0x7f1658102320 2024-04-26T20:31:32.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.327+0000 7f165f600700 1 -- 172.21.15.102:0/1204335686 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f165819a320 con 0x7f1658102320 2024-04-26T20:31:32.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.327+0000 7f1656000700 1 -- 172.21.15.102:0/1204335686 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f164800ce30 con 0x7f1658102320 2024-04-26T20:31:32.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.327+0000 7f1656000700 1 -- 172.21.15.102:0/1204335686 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1648010430 con 0x7f1658102320 2024-04-26T20:31:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.327+0000 7f165f600700 1 -- 172.21.15.102:0/1204335686 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1658103f30 con 0x7f1658102320 2024-04-26T20:31:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.328+0000 7f165f600700 1 -- 172.21.15.102:0/1204335686 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f165806a110 con 0x7f1658102320 2024-04-26T20:31:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.330+0000 7f1656000700 1 -- 172.21.15.102:0/1204335686 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f1648004000 con 0x7f1658102320 2024-04-26T20:31:32.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.332+0000 7f1656000700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f163c0a3c40 0x7f163c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:32.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.332+0000 7f1656000700 1 -- 172.21.15.102:0/1204335686 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f16480d01f0 con 0x7f1658102320 2024-04-26T20:31:32.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.332+0000 7f164fe00700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f163c0a3c40 0x7f163c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:32.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.333+0000 7f164fe00700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f163c0a3c40 0x7f163c0a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1640009fd0 tx=0x7f164001d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:32.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.336+0000 7f1656000700 1 -- 172.21.15.102:0/1204335686 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f164809cfd0 con 0x7f1658102320 2024-04-26T20:31:32.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.565+0000 7f165f600700 1 -- 172.21.15.102:0/1204335686 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1658064030 con 0x7f1658102320 2024-04-26T20:31:32.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.567+0000 7f1656000700 1 -- 172.21.15.102:0/1204335686 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18492 (secure 0 0 0) 0x7f164809c8d0 con 0x7f1658102320 2024-04-26T20:31:32.567 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:32.568 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:31:29.744387+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:31:32.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.569+0000 7f165f600700 1 -- 172.21.15.102:0/1204335686 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f163c0a3c40 msgr2=0x7f163c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:32.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.569+0000 7f165f600700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f163c0a3c40 0x7f163c0a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1640009fd0 tx=0x7f164001d040 comp rx=0 tx=0).stop 2024-04-26T20:31:32.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.570+0000 7f165f600700 1 -- 172.21.15.102:0/1204335686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1658102320 msgr2=0x7f165819f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:32.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.570+0000 7f165f600700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1658102320 0x7f165819f710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f164800ec10 tx=0x7f164800ef20 comp rx=0 tx=0).stop 2024-04-26T20:31:32.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.570+0000 7f165f600700 1 -- 172.21.15.102:0/1204335686 shutdown_connections 2024-04-26T20:31:32.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.570+0000 7f165f600700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f163c0a3c40 0x7f163c0a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:32.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.570+0000 7f165f600700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1658108390 0x7f16581998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:32.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.570+0000 7f165f600700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1658102320 0x7f165819f710 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:32.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.570+0000 7f165f600700 1 --2- 172.21.15.102:0/1204335686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1658102ce0 0x7f165819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:32.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.570+0000 7f165f600700 1 -- 172.21.15.102:0/1204335686 >> 172.21.15.102:0/1204335686 conn(0x7f16580792d0 msgr2=0x7f1658101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:32.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.570+0000 7f165f600700 1 -- 172.21.15.102:0/1204335686 shutdown_connections 2024-04-26T20:31:32.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:32.570+0000 7f165f600700 1 -- 172.21.15.102:0/1204335686 wait complete. 2024-04-26T20:31:33.422 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:27:10.527933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T20:31:33.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T20:31:33.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:33 smithi102 ceph-mon[28094]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:33.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:33 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1204335686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:31:33.866 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:31:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:33 smithi130 ceph-mon[32704]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:33 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1204335686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:31:34.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:33 smithi112 ceph-mon[32769]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:34.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:33 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1204335686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:31:35.704 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:35 smithi102 ceph-mon[28094]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:35.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:35 smithi130 ceph-mon[32704]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.997+0000 7f9dfd400700 1 -- 172.21.15.102:0/835778676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9df8105640 msgr2=0x7f9df8105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.997+0000 7f9dfd400700 1 --2- 172.21.15.102:0/835778676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9df8105640 0x7f9df8105ac0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009990 tx=0x7f9de4009ca0 comp rx=0 tx=0).stop 2024-04-26T20:31:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.997+0000 7f9dfd400700 1 -- 172.21.15.102:0/835778676 shutdown_connections 2024-04-26T20:31:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.997+0000 7f9dfd400700 1 --2- 172.21.15.102:0/835778676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df8106000 0x7f9df8112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.997+0000 7f9dfd400700 1 --2- 172.21.15.102:0/835778676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df810b640 0x7f9df810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:35.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.997+0000 7f9dfd400700 1 --2- 172.21.15.102:0/835778676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9df8105640 0x7f9df8105ac0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.997+0000 7f9dfd400700 1 -- 172.21.15.102:0/835778676 >> 172.21.15.102:0/835778676 conn(0x7f9df8101130 msgr2=0x7f9df8103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.997+0000 7f9dfd400700 1 -- 172.21.15.102:0/835778676 shutdown_connections 2024-04-26T20:31:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.997+0000 7f9dfd400700 1 -- 172.21.15.102:0/835778676 wait complete. 2024-04-26T20:31:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.998+0000 7f9dfd400700 1 Processor -- start 2024-04-26T20:31:35.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9dfd400700 1 -- start start 2024-04-26T20:31:36.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9dfd400700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9df8105640 0x7f9df819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:36.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9dfd400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df8106000 0x7f9df819c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9dfd400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df810b640 0x7f9df81a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9dfd400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9df81158d0 con 0x7f9df810b640 2024-04-26T20:31:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9dfd400700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9df8115750 con 0x7f9df8105640 2024-04-26T20:31:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9dfd400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9df8115a50 con 0x7f9df8106000 2024-04-26T20:31:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9df6800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9df8105640 0x7f9df819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9df6800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9df8105640 0x7f9df819bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:42676/0 (socket says 172.21.15.102:42676) 2024-04-26T20:31:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9df6800700 1 -- 172.21.15.102:0/2591293496 learned_addr learned my addr 172.21.15.102:0/2591293496 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:35.999+0000 7f9df7200700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df810b640 0x7f9df81a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:36.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.000+0000 7f9df5e00700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df8106000 0x7f9df819c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:36.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.000+0000 7f9df6800700 1 -- 172.21.15.102:0/2591293496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df8106000 msgr2=0x7f9df819c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:36.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.000+0000 7f9df6800700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df8106000 0x7f9df819c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:36.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.000+0000 7f9df6800700 1 -- 172.21.15.102:0/2591293496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df810b640 msgr2=0x7f9df81a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:36.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.000+0000 7f9df6800700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df810b640 0x7f9df81a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:36.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.000+0000 7f9df6800700 1 -- 172.21.15.102:0/2591293496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df81a0d50 con 0x7f9df8105640 2024-04-26T20:31:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.000+0000 7f9df6800700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9df8105640 0x7f9df819bd40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9de800e9c0 tx=0x7f9de800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.000+0000 7f9def400700 1 -- 172.21.15.102:0/2591293496 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de800cbd0 con 0x7f9df8105640 2024-04-26T20:31:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.001+0000 7f9def400700 1 -- 172.21.15.102:0/2591293496 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9de800ee50 con 0x7f9df8105640 2024-04-26T20:31:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.001+0000 7f9dfd400700 1 -- 172.21.15.102:0/2591293496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df81a1040 con 0x7f9df8105640 2024-04-26T20:31:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.001+0000 7f9def400700 1 -- 172.21.15.102:0/2591293496 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8005550 con 0x7f9df8105640 2024-04-26T20:31:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.001+0000 7f9dfd400700 1 -- 172.21.15.102:0/2591293496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df81a88c0 con 0x7f9df8105640 2024-04-26T20:31:36.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.002+0000 7f9dfd400700 1 -- 172.21.15.102:0/2591293496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df806a110 con 0x7f9df8105640 2024-04-26T20:31:36.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.004+0000 7f9def400700 1 -- 172.21.15.102:0/2591293496 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9de8004750 con 0x7f9df8105640 2024-04-26T20:31:36.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.005+0000 7f9def400700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9ddc0a3cc0 0x7f9ddc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:36.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.005+0000 7f9def400700 1 -- 172.21.15.102:0/2591293496 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9de80d0160 con 0x7f9df8105640 2024-04-26T20:31:36.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.005+0000 7f9df5e00700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9ddc0a3cc0 0x7f9ddc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:36.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.006+0000 7f9df5e00700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9ddc0a3cc0 0x7f9ddc0a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9df8103770 tx=0x7f9de400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:36.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.007+0000 7f9def400700 1 -- 172.21.15.102:0/2591293496 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9de809cf40 con 0x7f9df8105640 2024-04-26T20:31:36.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:35 smithi112 ceph-mon[32769]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.232+0000 7f9dfd400700 1 -- 172.21.15.102:0/2591293496 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9df819d0a0 con 0x7f9df8105640 2024-04-26T20:31:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.232+0000 7f9def400700 1 -- 172.21.15.102:0/2591293496 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f9de809c840 con 0x7f9df8105640 2024-04-26T20:31:36.234 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-26T20:31:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.235+0000 7f9dfd400700 1 -- 172.21.15.102:0/2591293496 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9ddc0a3cc0 msgr2=0x7f9ddc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.235+0000 7f9dfd400700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9ddc0a3cc0 0x7f9ddc0a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9df8103770 tx=0x7f9de400b260 comp rx=0 tx=0).stop 2024-04-26T20:31:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.235+0000 7f9dfd400700 1 -- 172.21.15.102:0/2591293496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9df8105640 msgr2=0x7f9df819bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.235+0000 7f9dfd400700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9df8105640 0x7f9df819bd40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9de800e9c0 tx=0x7f9de800ecd0 comp rx=0 tx=0).stop 2024-04-26T20:31:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.235+0000 7f9dfd400700 1 -- 172.21.15.102:0/2591293496 shutdown_connections 2024-04-26T20:31:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.236+0000 7f9dfd400700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9ddc0a3cc0 0x7f9ddc0a6180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.236+0000 7f9dfd400700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df810b640 0x7f9df81a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.236+0000 7f9dfd400700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9df8105640 0x7f9df819bd40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.236+0000 7f9dfd400700 1 --2- 172.21.15.102:0/2591293496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df8106000 0x7f9df819c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:36.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.236+0000 7f9dfd400700 1 -- 172.21.15.102:0/2591293496 >> 172.21.15.102:0/2591293496 conn(0x7f9df8101130 msgr2=0x7f9df81018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:36.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.236+0000 7f9dfd400700 1 -- 172.21.15.102:0/2591293496 shutdown_connections 2024-04-26T20:31:36.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:36.236+0000 7f9dfd400700 1 -- 172.21.15.102:0/2591293496 wait complete. 2024-04-26T20:31:36.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:36 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2591293496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:31:36.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:36 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2591293496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:31:37.005 INFO:tasks.cephadm:Adding mds.a on smithi102 2024-04-26T20:31:37.006 INFO:tasks.cephadm:Adding mds.d on smithi102 2024-04-26T20:31:37.006 INFO:tasks.cephadm:Adding mds.g on smithi102 2024-04-26T20:31:37.006 INFO:tasks.cephadm:Adding mds.j on smithi102 2024-04-26T20:31:37.006 INFO:tasks.cephadm:Adding mds.b on smithi112 2024-04-26T20:31:37.006 INFO:tasks.cephadm:Adding mds.e on smithi112 2024-04-26T20:31:37.006 INFO:tasks.cephadm:Adding mds.h on smithi112 2024-04-26T20:31:37.006 INFO:tasks.cephadm:Adding mds.k on smithi112 2024-04-26T20:31:37.006 INFO:tasks.cephadm:Adding mds.c on smithi130 2024-04-26T20:31:37.006 INFO:tasks.cephadm:Adding mds.f on smithi130 2024-04-26T20:31:37.006 INFO:tasks.cephadm:Adding mds.i on smithi130 2024-04-26T20:31:37.007 INFO:tasks.cephadm:Adding mds.l on smithi130 2024-04-26T20:31:37.007 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l' 2024-04-26T20:31:37.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:36 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2591293496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:31:37.277 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T20:31:37.672 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:37 smithi130 ceph-mon[32704]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:37.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:37 smithi102 ceph-mon[28094]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:38.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:37 smithi112 ceph-mon[32769]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:38.927 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.926+0000 7fed15aa4700 1 -- 172.21.15.130:0/2666464164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed10104800 msgr2=0x7fed10104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:38.927 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.926+0000 7fed15aa4700 1 --2- 172.21.15.130:0/2666464164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed10104800 0x7fed10104c60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fed04009990 tx=0x7fed04009ca0 comp rx=0 tx=0).stop 2024-04-26T20:31:38.927 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.926+0000 7fed15aa4700 1 -- 172.21.15.130:0/2666464164 shutdown_connections 2024-04-26T20:31:38.927 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.926+0000 7fed15aa4700 1 --2- 172.21.15.130:0/2666464164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed101051a0 0x7fed1010ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:38.927 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.926+0000 7fed15aa4700 1 --2- 172.21.15.130:0/2666464164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed10101820 0x7fed10101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:38.927 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.926+0000 7fed15aa4700 1 --2- 172.21.15.130:0/2666464164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed10104800 0x7fed10104c60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:38.927 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.926+0000 7fed15aa4700 1 -- 172.21.15.130:0/2666464164 >> 172.21.15.130:0/2666464164 conn(0x7fed100fd180 msgr2=0x7fed100ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:38.927 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.926+0000 7fed15aa4700 1 -- 172.21.15.130:0/2666464164 shutdown_connections 2024-04-26T20:31:38.927 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.926+0000 7fed15aa4700 1 -- 172.21.15.130:0/2666464164 wait complete. 2024-04-26T20:31:38.929 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.926+0000 7fed15aa4700 1 Processor -- start 2024-04-26T20:31:38.929 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed15aa4700 1 -- start start 2024-04-26T20:31:38.929 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed15aa4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed10101820 0x7fed1019b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:38.929 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed15aa4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed10104800 0x7fed1019be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:38.929 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed15aa4700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed101051a0 0x7fed101a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:38.929 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed15aa4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fed1010f0e0 con 0x7fed10101820 2024-04-26T20:31:38.929 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed15aa4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fed1010ef60 con 0x7fed10104800 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed15aa4700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fed1010f260 con 0x7fed101051a0 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed101051a0 0x7fed101a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed101051a0 0x7fed101a2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:55926/0 (socket says 172.21.15.130:55926) 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0ffff700 1 -- 172.21.15.130:0/391390514 learned_addr learned my addr 172.21.15.130:0/391390514 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0effd700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed10104800 0x7fed1019be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0ffff700 1 -- 172.21.15.130:0/391390514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed10104800 msgr2=0x7fed1019be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0ffff700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed10104800 0x7fed1019be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0ffff700 1 -- 172.21.15.130:0/391390514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed10101820 msgr2=0x7fed1019b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0ffff700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed10101820 0x7fed1019b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0ffff700 1 -- 172.21.15.130:0/391390514 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed101a3590 con 0x7fed101051a0 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0effd700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed10104800 0x7fed1019be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.927+0000 7fed0ffff700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed101051a0 0x7fed101a2ef0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fed0800b9a0 tx=0x7fed0800bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:38.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.928+0000 7fed0cff9700 1 -- 172.21.15.130:0/391390514 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08019070 con 0x7fed101051a0 2024-04-26T20:31:38.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.928+0000 7fed15aa4700 1 -- 172.21.15.130:0/391390514 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed101a3880 con 0x7fed101051a0 2024-04-26T20:31:38.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.928+0000 7fed0cff9700 1 -- 172.21.15.130:0/391390514 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed08014440 con 0x7fed101051a0 2024-04-26T20:31:38.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.928+0000 7fed0cff9700 1 -- 172.21.15.130:0/391390514 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0800ce60 con 0x7fed101051a0 2024-04-26T20:31:38.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.928+0000 7fed15aa4700 1 -- 172.21.15.130:0/391390514 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed101a3e40 con 0x7fed101051a0 2024-04-26T20:31:38.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.929+0000 7fed15aa4700 1 -- 172.21.15.130:0/391390514 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed1006a110 con 0x7fed101051a0 2024-04-26T20:31:38.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.930+0000 7fed0cff9700 1 -- 172.21.15.130:0/391390514 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fed0801c070 con 0x7fed101051a0 2024-04-26T20:31:38.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.930+0000 7fed0cff9700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fecf80a3c40 0x7fecf80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:38.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.930+0000 7fed0cff9700 1 -- 172.21.15.130:0/391390514 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fed080cf3a0 con 0x7fed101051a0 2024-04-26T20:31:38.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.931+0000 7fed0f7fe700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fecf80a3c40 0x7fecf80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:38.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.932+0000 7fed0f7fe700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fecf80a3c40 0x7fecf80a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fed1006b930 tx=0x7fed00006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:38.935 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:38.935+0000 7fed0cff9700 1 -- 172.21.15.130:0/391390514 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fed0809c2b0 con 0x7fed101051a0 2024-04-26T20:31:39.142 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.141+0000 7fed15aa4700 1 -- 172.21.15.130:0/391390514 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}) -- 0x7fed10103e10 con 0x7fecf80a3c40 2024-04-26T20:31:39.159 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.158+0000 7fed0cff9700 1 -- 172.21.15.130:0/391390514 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fed10103e10 con 0x7fecf80a3c40 2024-04-26T20:31:39.159 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mds.all update... 2024-04-26T20:31:39.161 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 -- 172.21.15.130:0/391390514 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fecf80a3c40 msgr2=0x7fecf80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:39.161 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fecf80a3c40 0x7fecf80a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fed1006b930 tx=0x7fed00006d20 comp rx=0 tx=0).stop 2024-04-26T20:31:39.161 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 -- 172.21.15.130:0/391390514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed101051a0 msgr2=0x7fed101a2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:39.161 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed101051a0 0x7fed101a2ef0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fed0800b9a0 tx=0x7fed0800bd60 comp rx=0 tx=0).stop 2024-04-26T20:31:39.161 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 -- 172.21.15.130:0/391390514 shutdown_connections 2024-04-26T20:31:39.161 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fecf80a3c40 0x7fecf80a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:39.161 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed101051a0 0x7fed101a2ef0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:39.162 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed10101820 0x7fed1019b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:39.162 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 --2- 172.21.15.130:0/391390514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed10104800 0x7fed1019be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:39.162 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 -- 172.21.15.130:0/391390514 >> 172.21.15.130:0/391390514 conn(0x7fed100fd180 msgr2=0x7fed101025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:39.162 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 -- 172.21.15.130:0/391390514 shutdown_connections 2024-04-26T20:31:39.162 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-26T20:31:39.160+0000 7fed15aa4700 1 -- 172.21.15.130:0/391390514 wait complete. 2024-04-26T20:31:39.803 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:39 smithi130 ceph-mon[32704]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:39.804 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:39.804 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:39 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:39.831 DEBUG:teuthology.orchestra.run.smithi102:mds.a> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T20:31:39.832 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:39 smithi102 ceph-mon[28094]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:39.832 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:39.833 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:39 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:39.836 DEBUG:teuthology.orchestra.run.smithi102:mds.d> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T20:31:39.881 DEBUG:teuthology.orchestra.run.smithi102:mds.g> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T20:31:39.928 DEBUG:teuthology.orchestra.run.smithi102:mds.j> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T20:31:39.974 DEBUG:teuthology.orchestra.run.smithi112:mds.b> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T20:31:39.977 DEBUG:teuthology.orchestra.run.smithi112:mds.e> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T20:31:39.996 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:39 smithi112 ceph-mon[32769]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:39.996 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:39.996 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:39 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:31:39.997 DEBUG:teuthology.orchestra.run.smithi112:mds.h> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T20:31:40.040 DEBUG:teuthology.orchestra.run.smithi112:mds.k> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T20:31:40.083 DEBUG:teuthology.orchestra.run.smithi130:mds.c> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T20:31:40.085 DEBUG:teuthology.orchestra.run.smithi130:mds.f> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T20:31:40.128 DEBUG:teuthology.orchestra.run.smithi130:mds.i> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T20:31:40.171 DEBUG:teuthology.orchestra.run.smithi130:mds.l> sudo journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T20:31:40.214 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T20:31:40.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:31:40.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.422+0000 7f98f8a00640 1 -- 172.21.15.102:0/3508294739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f4100fd0 msgr2=0x7f98f41013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.422+0000 7f98f8a00640 1 --2- 172.21.15.102:0/3508294739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f4100fd0 0x7f98f41013f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009900 tx=0x7f98e402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.423+0000 7f98f8a00640 1 -- 172.21.15.102:0/3508294739 shutdown_connections 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.423+0000 7f98f8a00640 1 --2- 172.21.15.102:0/3508294739 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98f41022d0 0x7f98f4102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.423+0000 7f98f8a00640 1 --2- 172.21.15.102:0/3508294739 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98f4101930 0x7f98f4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.423+0000 7f98f8a00640 1 --2- 172.21.15.102:0/3508294739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f4100fd0 0x7f98f41013f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.423+0000 7f98f8a00640 1 -- 172.21.15.102:0/3508294739 >> 172.21.15.102:0/3508294739 conn(0x7f98f40fcbf0 msgr2=0x7f98f40ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.423+0000 7f98f8a00640 1 -- 172.21.15.102:0/3508294739 shutdown_connections 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.423+0000 7f98f8a00640 1 -- 172.21.15.102:0/3508294739 wait complete. 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f8a00640 1 Processor -- start 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f8a00640 1 -- start start 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f8a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98f4100fd0 0x7f98f419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f8a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98f4101930 0x7f98f419b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f8a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f41022d0 0x7f98f41a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:40.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f8a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f98f4110450 con 0x7f98f4101930 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f8a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f98f41102d0 con 0x7f98f4100fd0 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f8a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f98f41105d0 con 0x7f98f41022d0 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f3400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98f4100fd0 0x7f98f419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f3400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98f4100fd0 0x7f98f419abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:42682/0 (socket says 172.21.15.102:42682) 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f3400640 1 -- 172.21.15.102:0/1132904685 learned_addr learned my addr 172.21.15.102:0/1132904685 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.424+0000 7f98f3e00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f41022d0 0x7f98f41a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98f2a00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98f4101930 0x7f98f419b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98f3e00640 1 -- 172.21.15.102:0/1132904685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98f4100fd0 msgr2=0x7f98f419abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98f3e00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98f4100fd0 0x7f98f419abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98f3e00640 1 -- 172.21.15.102:0/1132904685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98f4101930 msgr2=0x7f98f419b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98f3e00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98f4101930 0x7f98f419b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98f3e00640 1 -- 172.21.15.102:0/1132904685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f41a27d0 con 0x7f98f41022d0 2024-04-26T20:31:40.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98f3e00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f41022d0 0x7f98f41a2160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009fd0 tx=0x7f98ec00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:40.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98ebe00640 1 -- 172.21.15.102:0/1132904685 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec00ce20 con 0x7f98f41022d0 2024-04-26T20:31:40.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98ebe00640 1 -- 172.21.15.102:0/1132904685 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ec005020 con 0x7f98f41022d0 2024-04-26T20:31:40.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98f8a00640 1 -- 172.21.15.102:0/1132904685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98f41a2ac0 con 0x7f98f41022d0 2024-04-26T20:31:40.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98ebe00640 1 -- 172.21.15.102:0/1132904685 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec010690 con 0x7f98f41022d0 2024-04-26T20:31:40.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.425+0000 7f98f8a00640 1 -- 172.21.15.102:0/1132904685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f41a2fc0 con 0x7f98f41022d0 2024-04-26T20:31:40.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.427+0000 7f98f8a00640 1 -- 172.21.15.102:0/1132904685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98f410bd20 con 0x7f98f41022d0 2024-04-26T20:31:40.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.428+0000 7f98ebe00640 1 -- 172.21.15.102:0/1132904685 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f98ec0054d0 con 0x7f98f41022d0 2024-04-26T20:31:40.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.430+0000 7f98ebe00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98d80a3bb0 0x7f98d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:40.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.430+0000 7f98ebe00640 1 -- 172.21.15.102:0/1132904685 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f98ec0d06c0 con 0x7f98f41022d0 2024-04-26T20:31:40.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.430+0000 7f98f3400640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98d80a3bb0 0x7f98d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:40.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.430+0000 7f98f3400640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98d80a3bb0 0x7f98d80a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009230 tx=0x7f98e40058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:40.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.433+0000 7f98ebe00640 1 -- 172.21.15.102:0/1132904685 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f98ec09d4a0 con 0x7f98f41022d0 2024-04-26T20:31:40.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.639+0000 7f98f8a00640 1 -- 172.21.15.102:0/1132904685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98f4063530 con 0x7f98f41022d0 2024-04-26T20:31:40.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.640+0000 7f98ebe00640 1 -- 172.21.15.102:0/1132904685 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18492 (secure 0 0 0) 0x7f98ec09cda0 con 0x7f98f41022d0 2024-04-26T20:31:40.641 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:40.641 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:31:29.744387+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:31:40.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.642+0000 7f98f8a00640 1 -- 172.21.15.102:0/1132904685 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98d80a3bb0 msgr2=0x7f98d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:40.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.642+0000 7f98f8a00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98d80a3bb0 0x7f98d80a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009230 tx=0x7f98e40058d0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.642+0000 7f98f8a00640 1 -- 172.21.15.102:0/1132904685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f41022d0 msgr2=0x7f98f41a2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:40.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.642+0000 7f98f8a00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f41022d0 0x7f98f41a2160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009fd0 tx=0x7f98ec00ef10 comp rx=0 tx=0).stop 2024-04-26T20:31:40.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.643+0000 7f98f8a00640 1 -- 172.21.15.102:0/1132904685 shutdown_connections 2024-04-26T20:31:40.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.643+0000 7f98f8a00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98d80a3bb0 0x7f98d80a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.643+0000 7f98f8a00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98f4101930 0x7f98f419b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.643+0000 7f98f8a00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98f41022d0 0x7f98f41a2160 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.643+0000 7f98f8a00640 1 --2- 172.21.15.102:0/1132904685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98f4100fd0 0x7f98f419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.643+0000 7f98f8a00640 1 -- 172.21.15.102:0/1132904685 >> 172.21.15.102:0/1132904685 conn(0x7f98f40fcbf0 msgr2=0x7f98f40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:40.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.643+0000 7f98f8a00640 1 -- 172.21.15.102:0/1132904685 shutdown_connections 2024-04-26T20:31:40.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.643+0000 7f98f8a00640 1 -- 172.21.15.102:0/1132904685 wait complete. 2024-04-26T20:31:40.663 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:27:10.527933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T20:31:40.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:31:40.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.856+0000 7f9692000640 1 -- 172.21.15.102:0/2993361729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1028c0 msgr2=0x7f968c102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:40.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.856+0000 7f9692000640 1 --2- 172.21.15.102:0/2993361729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1028c0 0x7f968c102ce0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9680009900 tx=0x7f968002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.856+0000 7f9692000640 1 -- 172.21.15.102:0/2993361729 shutdown_connections 2024-04-26T20:31:40.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.856+0000 7f9692000640 1 --2- 172.21.15.102:0/2993361729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c103ab0 0x7f968c103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.856+0000 7f9692000640 1 --2- 172.21.15.102:0/2993361729 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c104470 0x7f968c10ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.856+0000 7f9692000640 1 --2- 172.21.15.102:0/2993361729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1028c0 0x7f968c102ce0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.856+0000 7f9692000640 1 -- 172.21.15.102:0/2993361729 >> 172.21.15.102:0/2993361729 conn(0x7f968c0fde30 msgr2=0x7f968c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:40.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.856+0000 7f9692000640 1 -- 172.21.15.102:0/2993361729 shutdown_connections 2024-04-26T20:31:40.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.856+0000 7f9692000640 1 -- 172.21.15.102:0/2993361729 wait complete. 2024-04-26T20:31:40.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.857+0000 7f9692000640 1 Processor -- start 2024-04-26T20:31:40.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9692000640 1 -- start start 2024-04-26T20:31:40.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9692000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1028c0 0x7f968c108640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:40.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9692000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c103ab0 0x7f968c108b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:40.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9692000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c104470 0x7f968c1051b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:40.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9692000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f968c10e3a0 con 0x7f968c103ab0 2024-04-26T20:31:40.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9692000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f968c10e220 con 0x7f968c104470 2024-04-26T20:31:40.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9692000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f968c10e520 con 0x7f968c1028c0 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9690c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1028c0 0x7f968c108640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9690c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1028c0 0x7f968c108640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:57782/0 (socket says 172.21.15.102:57782) 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9691600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c104470 0x7f968c1051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f9690c00640 1 -- 172.21.15.102:0/330057077 learned_addr learned my addr 172.21.15.102:0/330057077 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.858+0000 7f968be00640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c103ab0 0x7f968c108b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9691600640 1 -- 172.21.15.102:0/330057077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1028c0 msgr2=0x7f968c108640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9691600640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1028c0 0x7f968c108640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9691600640 1 -- 172.21.15.102:0/330057077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c103ab0 msgr2=0x7f968c108b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9691600640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c103ab0 0x7f968c108b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9691600640 1 -- 172.21.15.102:0/330057077 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968c105a70 con 0x7f968c104470 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9691600640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c104470 0x7f968c1051b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f968400ec50 tx=0x7f968400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9689600640 1 -- 172.21.15.102:0/330057077 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9684003b70 con 0x7f968c104470 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9689600640 1 -- 172.21.15.102:0/330057077 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9684003d10 con 0x7f968c104470 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9692000640 1 -- 172.21.15.102:0/330057077 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f968c1a7130 con 0x7f968c104470 2024-04-26T20:31:40.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9689600640 1 -- 172.21.15.102:0/330057077 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9684010830 con 0x7f968c104470 2024-04-26T20:31:40.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.859+0000 7f9692000640 1 -- 172.21.15.102:0/330057077 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f968c1a7540 con 0x7f968c104470 2024-04-26T20:31:40.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.861+0000 7f9692000640 1 -- 172.21.15.102:0/330057077 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f968c1a77d0 con 0x7f968c104470 2024-04-26T20:31:40.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.862+0000 7f9689600640 1 -- 172.21.15.102:0/330057077 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9684003ec0 con 0x7f968c104470 2024-04-26T20:31:40.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.863+0000 7f9689600640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96740a3bb0 0x7f96740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:40.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.863+0000 7f9689600640 1 -- 172.21.15.102:0/330057077 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f96840d0600 con 0x7f968c104470 2024-04-26T20:31:40.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.864+0000 7f9690c00640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96740a3bb0 0x7f96740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:40.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.864+0000 7f9690c00640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96740a3bb0 0x7f96740a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9680000c00 tx=0x7f9680005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:40.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:40.868+0000 7f9689600640 1 -- 172.21.15.102:0/330057077 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f968409d3e0 con 0x7f968c104470 2024-04-26T20:31:40.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:40 smithi130 ceph-mon[32704]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:31:40.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:40 smithi130 ceph-mon[32704]: Saving service mds.all spec with placement smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-26T20:31:40.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:40.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:40.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:40.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:40.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:40.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:40.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:40 smithi102 ceph-mon[28094]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:31:40.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:40 smithi102 ceph-mon[28094]: Saving service mds.all spec with placement smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-26T20:31:40.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:40.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:40.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:40.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:40.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:40.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:41.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:40 smithi112 ceph-mon[32769]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:31:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:40 smithi112 ceph-mon[32769]: Saving service mds.all spec with placement smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-26T20:31:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:31:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:41.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.069+0000 7f9692000640 1 -- 172.21.15.102:0/330057077 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f968c1a79a0 con 0x7f968c104470 2024-04-26T20:31:41.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.070+0000 7f9689600640 1 -- 172.21.15.102:0/330057077 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f968409cce0 con 0x7f968c104470 2024-04-26T20:31:41.071 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-26T20:31:41.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 -- 172.21.15.102:0/330057077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96740a3bb0 msgr2=0x7f96740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:41.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96740a3bb0 0x7f96740a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9680000c00 tx=0x7f9680005a20 comp rx=0 tx=0).stop 2024-04-26T20:31:41.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 -- 172.21.15.102:0/330057077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c104470 msgr2=0x7f968c1051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:41.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c104470 0x7f968c1051b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f968400ec50 tx=0x7f968400c6a0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 -- 172.21.15.102:0/330057077 shutdown_connections 2024-04-26T20:31:41.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96740a3bb0 0x7f96740a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f968c103ab0 0x7f968c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c104470 0x7f968c1051b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 --2- 172.21.15.102:0/330057077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1028c0 0x7f968c108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 -- 172.21.15.102:0/330057077 >> 172.21.15.102:0/330057077 conn(0x7f968c0fde30 msgr2=0x7f968c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:41.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 -- 172.21.15.102:0/330057077 shutdown_connections 2024-04-26T20:31:41.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.073+0000 7f9692000640 1 -- 172.21.15.102:0/330057077 wait complete. 2024-04-26T20:31:41.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:31:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.294+0000 7f40b0800640 1 -- 172.21.15.102:0/4007753991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f40a810ab40 msgr2=0x7f40a81113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.294+0000 7f40b0800640 1 --2- 172.21.15.102:0/4007753991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f40a810ab40 0x7f40a81113d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f40a4009ca0 tx=0x7f40a402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.294+0000 7f40b0800640 1 -- 172.21.15.102:0/4007753991 shutdown_connections 2024-04-26T20:31:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.294+0000 7f40b0800640 1 --2- 172.21.15.102:0/4007753991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f40a810ab40 0x7f40a81113d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.294+0000 7f40b0800640 1 --2- 172.21.15.102:0/4007753991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40a810a180 0x7f40a810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.294+0000 7f40b0800640 1 --2- 172.21.15.102:0/4007753991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40a8108f90 0x7f40a81093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.294+0000 7f40b0800640 1 -- 172.21.15.102:0/4007753991 >> 172.21.15.102:0/4007753991 conn(0x7f40a8077f40 msgr2=0x7f40a807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.295+0000 7f40b0800640 1 -- 172.21.15.102:0/4007753991 shutdown_connections 2024-04-26T20:31:41.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.295+0000 7f40b0800640 1 -- 172.21.15.102:0/4007753991 wait complete. 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.295+0000 7f40b0800640 1 Processor -- start 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40b0800640 1 -- start start 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40b0800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f40a8108f90 0x7f40a819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40b0800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40a810a180 0x7f40a819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40b0800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40a810ab40 0x7f40a81a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40b0800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f40a8114b20 con 0x7f40a810a180 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40b0800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f40a81149a0 con 0x7f40a8108f90 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40b0800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f40a8114ca0 con 0x7f40a810ab40 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40ad400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40a810a180 0x7f40a819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40ad400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40a810a180 0x7f40a819f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58808/0 (socket says 172.21.15.102:58808) 2024-04-26T20:31:41.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40ade00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f40a8108f90 0x7f40a819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40ad400640 1 -- 172.21.15.102:0/3490872126 learned_addr learned my addr 172.21.15.102:0/3490872126 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.296+0000 7f40ae800640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40a810ab40 0x7f40a81a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40ad400640 1 -- 172.21.15.102:0/3490872126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40a810ab40 msgr2=0x7f40a81a66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40ad400640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40a810ab40 0x7f40a81a66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40ad400640 1 -- 172.21.15.102:0/3490872126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f40a8108f90 msgr2=0x7f40a819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40ad400640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f40a8108f90 0x7f40a819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40ad400640 1 -- 172.21.15.102:0/3490872126 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a81a6d30 con 0x7f40a810a180 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40ade00640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f40a8108f90 0x7f40a819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40ad400640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40a810a180 0x7f40a819f650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f409800e9f0 tx=0x7f409800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40a2a00640 1 -- 172.21.15.102:0/3490872126 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409800cd80 con 0x7f40a810a180 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40a2a00640 1 -- 172.21.15.102:0/3490872126 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40980054d0 con 0x7f40a810a180 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40b0800640 1 -- 172.21.15.102:0/3490872126 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a81a6fc0 con 0x7f40a810a180 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40a2a00640 1 -- 172.21.15.102:0/3490872126 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098010690 con 0x7f40a810a180 2024-04-26T20:31:41.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.297+0000 7f40b0800640 1 -- 172.21.15.102:0/3490872126 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a81a7490 con 0x7f40a810a180 2024-04-26T20:31:41.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.298+0000 7f40b0800640 1 -- 172.21.15.102:0/3490872126 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a81102b0 con 0x7f40a810a180 2024-04-26T20:31:41.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.302+0000 7f40a2a00640 1 -- 172.21.15.102:0/3490872126 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f4098004e30 con 0x7f40a810a180 2024-04-26T20:31:41.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.303+0000 7f40a2a00640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f40900a3bb0 0x7f40900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:41.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.304+0000 7f40a2a00640 1 -- 172.21.15.102:0/3490872126 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f40980d05d0 con 0x7f40a810a180 2024-04-26T20:31:41.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.304+0000 7f40ade00640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f40900a3bb0 0x7f40900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:41.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.304+0000 7f40ade00640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f40900a3bb0 0x7f40900a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f40a8109fe0 tx=0x7f409c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:41.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.306+0000 7f40a2a00640 1 -- 172.21.15.102:0/3490872126 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4098054060 con 0x7f40a810a180 2024-04-26T20:31:41.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.550+0000 7f40b0800640 1 -- 172.21.15.102:0/3490872126 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40a802dad0 con 0x7f40a810a180 2024-04-26T20:31:41.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.551+0000 7f40a2a00640 1 -- 172.21.15.102:0/3490872126 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f409809d3b0 con 0x7f40a810a180 2024-04-26T20:31:41.552 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:41.552 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-26T20:31:41.552 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 1 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.553+0000 7f40b0800640 1 -- 172.21.15.102:0/3490872126 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f40900a3bb0 msgr2=0x7f40900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.553+0000 7f40b0800640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f40900a3bb0 0x7f40900a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f40a8109fe0 tx=0x7f409c005f90 comp rx=0 tx=0).stop 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.553+0000 7f40b0800640 1 -- 172.21.15.102:0/3490872126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40a810a180 msgr2=0x7f40a819f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.553+0000 7f40b0800640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40a810a180 0x7f40a819f650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f409800e9f0 tx=0x7f409800edf0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.554+0000 7f40b0800640 1 -- 172.21.15.102:0/3490872126 shutdown_connections 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.554+0000 7f40b0800640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f40900a3bb0 0x7f40900a6070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.554+0000 7f40b0800640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40a810a180 0x7f40a819f650 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.554+0000 7f40b0800640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f40a810ab40 0x7f40a81a66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.554+0000 7f40b0800640 1 --2- 172.21.15.102:0/3490872126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f40a8108f90 0x7f40a819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.554+0000 7f40b0800640 1 -- 172.21.15.102:0/3490872126 >> 172.21.15.102:0/3490872126 conn(0x7f40a8077f40 msgr2=0x7f40a810d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:41.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.554+0000 7f40b0800640 1 -- 172.21.15.102:0/3490872126 shutdown_connections 2024-04-26T20:31:41.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:41.554+0000 7f40b0800640 1 -- 172.21.15.102:0/3490872126 wait complete. 2024-04-26T20:31:41.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:41 smithi130 ceph-mon[32704]: Deploying daemon mds.l on smithi130 2024-04-26T20:31:41.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:41 smithi130 ceph-mon[32704]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:41.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:41 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1132904685' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:31:41.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:41 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/330057077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:31:41.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:41 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3490872126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:41.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:41 smithi102 ceph-mon[28094]: Deploying daemon mds.l on smithi130 2024-04-26T20:31:41.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:41 smithi102 ceph-mon[28094]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:41.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:41 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1132904685' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:31:41.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:41 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/330057077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:31:41.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:41 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3490872126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:42.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:41 smithi112 ceph-mon[32769]: Deploying daemon mds.l on smithi130 2024-04-26T20:31:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:41 smithi112 ceph-mon[32769]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:41 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1132904685' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:31:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:41 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/330057077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:31:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:41 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3490872126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:42.872 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:31:42 smithi130 podman[62711]: 2024-04-26 20:31:42.856619406 +0000 UTC m=+0.137785305 container create 5e5f8f74aad5b81fe5d7ca91018c2bace7ffb62cbc8d056101f366eb1fa766ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:31:43.372 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:31:43 smithi130 podman[62711]: 2024-04-26 20:31:43.109343263 +0000 UTC m=+0.390509162 container init 5e5f8f74aad5b81fe5d7ca91018c2bace7ffb62cbc8d056101f366eb1fa766ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-26T20:31:43.372 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:31:43 smithi130 podman[62711]: 2024-04-26 20:31:43.111795153 +0000 UTC m=+0.392961066 container start 5e5f8f74aad5b81fe5d7ca91018c2bace7ffb62cbc8d056101f366eb1fa766ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T20:31:43.372 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mds[62730]: -- 172.21.15.130:0/560926902 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f238503a40 con 0x55f2384d6c00 2024-04-26T20:31:43.372 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l[62726]: starting mds.l at 2024-04-26T20:31:43.372 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:31:43 smithi130 bash[62711]: 5e5f8f74aad5b81fe5d7ca91018c2bace7ffb62cbc8d056101f366eb1fa766ad 2024-04-26T20:31:43.372 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:31:43 smithi130 systemd[1]: Started Ceph mds.l for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:31:43.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:31:43.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.781+0000 7f9522800640 1 -- 172.21.15.102:0/795079261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f951c104b90 msgr2=0x7f951c104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:43.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.781+0000 7f9522800640 1 --2- 172.21.15.102:0/795079261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f951c104b90 0x7f951c104fb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f950c009900 tx=0x7f950c02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:31:43.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.782+0000 7f9522800640 1 -- 172.21.15.102:0/795079261 shutdown_connections 2024-04-26T20:31:43.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.782+0000 7f9522800640 1 --2- 172.21.15.102:0/795079261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c106740 0x7f951c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:43.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.782+0000 7f9522800640 1 --2- 172.21.15.102:0/795079261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f951c105d80 0x7f951c106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:43.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.782+0000 7f9522800640 1 --2- 172.21.15.102:0/795079261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f951c104b90 0x7f951c104fb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:43.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.782+0000 7f9522800640 1 -- 172.21.15.102:0/795079261 >> 172.21.15.102:0/795079261 conn(0x7f951c100120 msgr2=0x7f951c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.782+0000 7f9522800640 1 -- 172.21.15.102:0/795079261 shutdown_connections 2024-04-26T20:31:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.782+0000 7f9522800640 1 -- 172.21.15.102:0/795079261 wait complete. 2024-04-26T20:31:43.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.783+0000 7f9522800640 1 Processor -- start 2024-04-26T20:31:43.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.783+0000 7f9522800640 1 -- start start 2024-04-26T20:31:43.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.783+0000 7f9522800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f951c104b90 0x7f951c19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:43.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.783+0000 7f9522800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f951c105d80 0x7f951c19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:43.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.783+0000 7f9522800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c106740 0x7f951c1a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:43.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.783+0000 7f9522800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f951c1106e0 con 0x7f951c104b90 2024-04-26T20:31:43.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f9522800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f951c110560 con 0x7f951c105d80 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f9522800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f951c110860 con 0x7f951c106740 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f951c104b90 0x7f951c19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f951c105d80 0x7f951c19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f951c105d80 0x7f951c19b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:42736/0 (socket says 172.21.15.102:42736) 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951aa00640 1 -- 172.21.15.102:0/4105297138 learned_addr learned my addr 172.21.15.102:0/4105297138 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c106740 0x7f951c1a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f951c104b90 0x7f951c19adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58830/0 (socket says 172.21.15.102:58830) 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951aa00640 1 -- 172.21.15.102:0/4105297138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c106740 msgr2=0x7f951c1a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951aa00640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c106740 0x7f951c1a2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951aa00640 1 -- 172.21.15.102:0/4105297138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f951c104b90 msgr2=0x7f951c19adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951aa00640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f951c104b90 0x7f951c19adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951aa00640 1 -- 172.21.15.102:0/4105297138 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951c1a29d0 con 0x7f951c105d80 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951be00640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c106740 0x7f951c1a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:31:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.784+0000 7f951aa00640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f951c105d80 0x7f951c19b2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f950800ea60 tx=0x7f950800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.785+0000 7f9513e00640 1 -- 172.21.15.102:0/4105297138 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950800cde0 con 0x7f951c105d80 2024-04-26T20:31:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.785+0000 7f9513e00640 1 -- 172.21.15.102:0/4105297138 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95080035f0 con 0x7f951c105d80 2024-04-26T20:31:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.785+0000 7f9513e00640 1 -- 172.21.15.102:0/4105297138 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9508010690 con 0x7f951c105d80 2024-04-26T20:31:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.785+0000 7f9522800640 1 -- 172.21.15.102:0/4105297138 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f951c1a2cc0 con 0x7f951c105d80 2024-04-26T20:31:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.785+0000 7f9522800640 1 -- 172.21.15.102:0/4105297138 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f951c1a31c0 con 0x7f951c105d80 2024-04-26T20:31:43.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.786+0000 7f9522800640 1 -- 172.21.15.102:0/4105297138 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f951c10bf00 con 0x7f951c105d80 2024-04-26T20:31:43.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.789+0000 7f9513e00640 1 -- 172.21.15.102:0/4105297138 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f95080037a0 con 0x7f951c105d80 2024-04-26T20:31:43.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.790+0000 7f9513e00640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95000a3c30 0x7f95000a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:43.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.790+0000 7f9513e00640 1 -- 172.21.15.102:0/4105297138 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f95080d0240 con 0x7f951c105d80 2024-04-26T20:31:43.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.790+0000 7f951b400640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95000a3c30 0x7f95000a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:43.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.791+0000 7f951b400640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95000a3c30 0x7f95000a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f950c005ab0 tx=0x7f950c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:43.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:43.794+0000 7f9513e00640 1 -- 172.21.15.102:0/4105297138 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f950809d020 con 0x7f951c105d80 2024-04-26T20:31:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mon[32704]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.130:6832/4251888549,v1:172.21.15.130:6833/4251888549] up:boot 2024-04-26T20:31:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mon[32704]: fsmap 1 up:standby 2024-04-26T20:31:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T20:31:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:43.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:43.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:43 smithi102 ceph-mon[28094]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:43 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.130:6832/4251888549,v1:172.21.15.130:6833/4251888549] up:boot 2024-04-26T20:31:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:43 smithi102 ceph-mon[28094]: fsmap 1 up:standby 2024-04-26T20:31:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T20:31:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:43.939 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:43 smithi112 ceph-mon[32769]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:43.939 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:43 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.130:6832/4251888549,v1:172.21.15.130:6833/4251888549] up:boot 2024-04-26T20:31:43.939 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:43 smithi112 ceph-mon[32769]: fsmap 1 up:standby 2024-04-26T20:31:43.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T20:31:43.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:43.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:43.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:43.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:43.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:43.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.040+0000 7f9522800640 1 -- 172.21.15.102:0/4105297138 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f951c02dad0 con 0x7f951c105d80 2024-04-26T20:31:44.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.041+0000 7f9513e00640 1 -- 172.21.15.102:0/4105297138 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f950809c920 con 0x7f951c105d80 2024-04-26T20:31:44.042 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:44.042 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T20:31:44.042 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 2 2024-04-26T20:31:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.043+0000 7f9522800640 1 -- 172.21.15.102:0/4105297138 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95000a3c30 msgr2=0x7f95000a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.043+0000 7f9522800640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95000a3c30 0x7f95000a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f950c005ab0 tx=0x7f950c005a20 comp rx=0 tx=0).stop 2024-04-26T20:31:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.043+0000 7f9522800640 1 -- 172.21.15.102:0/4105297138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f951c105d80 msgr2=0x7f951c19b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.043+0000 7f9522800640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f951c105d80 0x7f951c19b2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f950800ea60 tx=0x7f950800ef20 comp rx=0 tx=0).stop 2024-04-26T20:31:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.044+0000 7f9522800640 1 -- 172.21.15.102:0/4105297138 shutdown_connections 2024-04-26T20:31:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.044+0000 7f9522800640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95000a3c30 0x7f95000a60f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.044+0000 7f9522800640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c106740 0x7f951c1a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.044+0000 7f9522800640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f951c105d80 0x7f951c19b2f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.044+0000 7f9522800640 1 --2- 172.21.15.102:0/4105297138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f951c104b90 0x7f951c19adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:44.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.044+0000 7f9522800640 1 -- 172.21.15.102:0/4105297138 >> 172.21.15.102:0/4105297138 conn(0x7f951c100120 msgr2=0x7f951c1090d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:44.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.044+0000 7f9522800640 1 -- 172.21.15.102:0/4105297138 shutdown_connections 2024-04-26T20:31:44.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:44.044+0000 7f9522800640 1 -- 172.21.15.102:0/4105297138 wait complete. 2024-04-26T20:31:44.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:44 smithi130 ceph-mon[32704]: Deploying daemon mds.k on smithi112 2024-04-26T20:31:44.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:44 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/4105297138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:44.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:44 smithi102 ceph-mon[28094]: Deploying daemon mds.k on smithi112 2024-04-26T20:31:44.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:44 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/4105297138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:44.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:44 smithi112 ceph-mon[32769]: Deploying daemon mds.k on smithi112 2024-04-26T20:31:44.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:44 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/4105297138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:45.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:45 smithi112 ceph-mon[32769]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:45.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:45 smithi130 ceph-mon[32704]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:45.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:45 smithi102 ceph-mon[28094]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:46.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:31:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.278+0000 7f70b0a00640 1 -- 172.21.15.102:0/678417461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ac103980 msgr2=0x7f70ac103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.278+0000 7f70b0a00640 1 --2- 172.21.15.102:0/678417461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ac103980 0x7f70ac103e00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7098009860 tx=0x7f709802a340 comp rx=0 tx=0).stop 2024-04-26T20:31:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.278+0000 7f70b0a00640 1 -- 172.21.15.102:0/678417461 shutdown_connections 2024-04-26T20:31:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.278+0000 7f70b0a00640 1 --2- 172.21.15.102:0/678417461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ac103980 0x7f70ac103e00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.278+0000 7f70b0a00640 1 --2- 172.21.15.102:0/678417461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70ac104340 0x7f70ac10abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.278+0000 7f70b0a00640 1 --2- 172.21.15.102:0/678417461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70ac102790 0x7f70ac102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.278+0000 7f70b0a00640 1 -- 172.21.15.102:0/678417461 >> 172.21.15.102:0/678417461 conn(0x7f70ac0fdd00 msgr2=0x7f70ac100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.278+0000 7f70b0a00640 1 -- 172.21.15.102:0/678417461 shutdown_connections 2024-04-26T20:31:46.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.278+0000 7f70b0a00640 1 -- 172.21.15.102:0/678417461 wait complete. 2024-04-26T20:31:46.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.279+0000 7f70b0a00640 1 Processor -- start 2024-04-26T20:31:46.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70b0a00640 1 -- start start 2024-04-26T20:31:46.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70b0a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70ac102790 0x7f70ac19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:46.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70b0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ac103980 0x7f70ac19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:46.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70b0a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70ac104340 0x7f70ac1a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:46.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70b0a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac10e2c0 con 0x7f70ac103980 2024-04-26T20:31:46.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70b0a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac10e140 con 0x7f70ac104340 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70b0a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac10e440 con 0x7f70ac102790 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ac103980 0x7f70ac19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70abe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70ac104340 0x7f70ac1a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ac103980 0x7f70ac19b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37734/0 (socket says 172.21.15.102:37734) 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70abe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70ac104340 0x7f70ac1a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:47996/0 (socket says 172.21.15.102:47996) 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70aaa00640 1 -- 172.21.15.102:0/2130007171 learned_addr learned my addr 172.21.15.102:0/2130007171 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.280+0000 7f70ab400640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70ac102790 0x7f70ac19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.281+0000 7f70aaa00640 1 -- 172.21.15.102:0/2130007171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70ac102790 msgr2=0x7f70ac19aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.281+0000 7f70aaa00640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70ac102790 0x7f70ac19aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.281+0000 7f70aaa00640 1 -- 172.21.15.102:0/2130007171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70ac104340 msgr2=0x7f70ac1a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.281+0000 7f70aaa00640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70ac104340 0x7f70ac1a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.281+0000 7f70aaa00640 1 -- 172.21.15.102:0/2130007171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ac1a27c0 con 0x7f70ac103980 2024-04-26T20:31:46.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.281+0000 7f70aaa00640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ac103980 0x7f70ac19b0e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f70980057c0 tx=0x7f7098032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:46.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.281+0000 7f70a3e00640 1 -- 172.21.15.102:0/2130007171 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098037040 con 0x7f70ac103980 2024-04-26T20:31:46.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.282+0000 7f70b0a00640 1 -- 172.21.15.102:0/2130007171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70ac1a2a50 con 0x7f70ac103980 2024-04-26T20:31:46.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.282+0000 7f70a3e00640 1 -- 172.21.15.102:0/2130007171 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7098030bd0 con 0x7f70ac103980 2024-04-26T20:31:46.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.282+0000 7f70a3e00640 1 -- 172.21.15.102:0/2130007171 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098005280 con 0x7f70ac103980 2024-04-26T20:31:46.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.282+0000 7f70b0a00640 1 -- 172.21.15.102:0/2130007171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70ac1a2ee0 con 0x7f70ac103980 2024-04-26T20:31:46.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.283+0000 7f70b0a00640 1 -- 172.21.15.102:0/2130007171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70ac1a31d0 con 0x7f70ac103980 2024-04-26T20:31:46.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.285+0000 7f70a3e00640 1 -- 172.21.15.102:0/2130007171 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f7098043020 con 0x7f70ac103980 2024-04-26T20:31:46.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.287+0000 7f70a3e00640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f70900a3bb0 0x7f70900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:46.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.287+0000 7f70a3e00640 1 -- 172.21.15.102:0/2130007171 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f70980efb30 con 0x7f70ac103980 2024-04-26T20:31:46.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.288+0000 7f70ab400640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f70900a3bb0 0x7f70900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:46.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.288+0000 7f70ab400640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f70900a3bb0 0x7f70900a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f709c009530 tx=0x7f709c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:46.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.290+0000 7f70a3e00640 1 -- 172.21.15.102:0/2130007171 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f70980bc910 con 0x7f70ac103980 2024-04-26T20:31:46.527 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:31:46 smithi112 systemd[1]: Started Ceph mds.k for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:31:46.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.535+0000 7f70b0a00640 1 -- 172.21.15.102:0/2130007171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70ac1a33a0 con 0x7f70ac103980 2024-04-26T20:31:46.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.536+0000 7f70a3e00640 1 -- 172.21.15.102:0/2130007171 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f70980bc210 con 0x7f70ac103980 2024-04-26T20:31:46.539 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:46.539 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T20:31:46.539 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 2 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.538+0000 7f70b0a00640 1 -- 172.21.15.102:0/2130007171 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f70900a3bb0 msgr2=0x7f70900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.538+0000 7f70b0a00640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f70900a3bb0 0x7f70900a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f709c009530 tx=0x7f709c009260 comp rx=0 tx=0).stop 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.538+0000 7f70b0a00640 1 -- 172.21.15.102:0/2130007171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ac103980 msgr2=0x7f70ac19b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.538+0000 7f70b0a00640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ac103980 0x7f70ac19b0e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f70980057c0 tx=0x7f7098032da0 comp rx=0 tx=0).stop 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.539+0000 7f70b0a00640 1 -- 172.21.15.102:0/2130007171 shutdown_connections 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.539+0000 7f70b0a00640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f70900a3bb0 0x7f70900a6070 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.539+0000 7f70b0a00640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70ac103980 0x7f70ac19b0e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.539+0000 7f70b0a00640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70ac104340 0x7f70ac1a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.539+0000 7f70b0a00640 1 --2- 172.21.15.102:0/2130007171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70ac102790 0x7f70ac19aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.539+0000 7f70b0a00640 1 -- 172.21.15.102:0/2130007171 >> 172.21.15.102:0/2130007171 conn(0x7f70ac0fdd00 msgr2=0x7f70ac074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.539+0000 7f70b0a00640 1 -- 172.21.15.102:0/2130007171 shutdown_connections 2024-04-26T20:31:46.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:46.539+0000 7f70b0a00640 1 -- 172.21.15.102:0/2130007171 wait complete. 2024-04-26T20:31:47.494 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: Deploying daemon mds.j on smithi102 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2130007171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.112:6832/1543441778,v1:172.21.15.112:6833/1543441778] up:boot 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: fsmap 2 up:standby 2024-04-26T20:31:47.495 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T20:31:47.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:47.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:47.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:47.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:47.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:47.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:47.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: Deploying daemon mds.j on smithi102 2024-04-26T20:31:47.529 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:47.530 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2130007171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:47.530 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.112:6832/1543441778,v1:172.21.15.112:6833/1543441778] up:boot 2024-04-26T20:31:47.530 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: fsmap 2 up:standby 2024-04-26T20:31:47.530 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T20:31:47.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: Deploying daemon mds.j on smithi102 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2130007171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.112:6832/1543441778,v1:172.21.15.112:6833/1543441778] up:boot 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: fsmap 2 up:standby 2024-04-26T20:31:47.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T20:31:48.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:31:48.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.994+0000 7f4e40600640 1 -- 172.21.15.102:0/1062402216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e38102980 msgr2=0x7f4e38102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:48.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.994+0000 7f4e40600640 1 --2- 172.21.15.102:0/1062402216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e38102980 0x7f4e38102da0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009900 tx=0x7f4e2402a410 comp rx=0 tx=0).stop 2024-04-26T20:31:48.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.994+0000 7f4e40600640 1 -- 172.21.15.102:0/1062402216 shutdown_connections 2024-04-26T20:31:48.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.994+0000 7f4e40600640 1 --2- 172.21.15.102:0/1062402216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e38104530 0x7f4e3810adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:48.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.994+0000 7f4e40600640 1 --2- 172.21.15.102:0/1062402216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e38103b70 0x7f4e38103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:48.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.994+0000 7f4e40600640 1 --2- 172.21.15.102:0/1062402216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e38102980 0x7f4e38102da0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:48.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.994+0000 7f4e40600640 1 -- 172.21.15.102:0/1062402216 >> 172.21.15.102:0/1062402216 conn(0x7f4e380fdf10 msgr2=0x7f4e38100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:48.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.994+0000 7f4e40600640 1 -- 172.21.15.102:0/1062402216 shutdown_connections 2024-04-26T20:31:48.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.994+0000 7f4e40600640 1 -- 172.21.15.102:0/1062402216 wait complete. 2024-04-26T20:31:48.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.996+0000 7f4e40600640 1 Processor -- start 2024-04-26T20:31:48.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.997+0000 7f4e40600640 1 -- start start 2024-04-26T20:31:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.997+0000 7f4e40600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e38102980 0x7f4e38198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.997+0000 7f4e40600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e38103b70 0x7f4e381990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.997+0000 7f4e40600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e38104530 0x7f4e381a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.997+0000 7f4e40600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3810e4f0 con 0x7f4e38103b70 2024-04-26T20:31:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.997+0000 7f4e40600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3810e370 con 0x7f4e38102980 2024-04-26T20:31:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.997+0000 7f4e40600640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3810e670 con 0x7f4e38104530 2024-04-26T20:31:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.998+0000 7f4e3dc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e38102980 0x7f4e38198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.998+0000 7f4e3dc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e38102980 0x7f4e38198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:48016/0 (socket says 172.21.15.102:48016) 2024-04-26T20:31:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.998+0000 7f4e3dc00640 1 -- 172.21.15.102:0/908493730 learned_addr learned my addr 172.21.15.102:0/908493730 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.998+0000 7f4e3e600640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e38104530 0x7f4e381a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.998+0000 7f4e3dc00640 1 -- 172.21.15.102:0/908493730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e38104530 msgr2=0x7f4e381a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.998+0000 7f4e3dc00640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e38104530 0x7f4e381a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:48.998+0000 7f4e3dc00640 1 -- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e38103b70 msgr2=0x7f4e381990b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:31:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.002+0000 7f4e3d200640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e38103b70 0x7f4e381990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.002+0000 7f4e3dc00640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e38103b70 0x7f4e381990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.002+0000 7f4e3dc00640 1 -- 172.21.15.102:0/908493730 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e381a0790 con 0x7f4e38102980 2024-04-26T20:31:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.002+0000 7f4e3e600640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e38104530 0x7f4e381a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:31:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.002+0000 7f4e3d200640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e38103b70 0x7f4e381990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T20:31:49.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.002+0000 7f4e3dc00640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e38102980 0x7f4e38198b70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e24005ab0 tx=0x7f4e24003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.003+0000 7f4e36000640 1 -- 172.21.15.102:0/908493730 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2403f070 con 0x7f4e38102980 2024-04-26T20:31:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.003+0000 7f4e36000640 1 -- 172.21.15.102:0/908493730 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e24031c60 con 0x7f4e38102980 2024-04-26T20:31:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.003+0000 7f4e40600640 1 -- 172.21.15.102:0/908493730 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e381a0a20 con 0x7f4e38102980 2024-04-26T20:31:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.003+0000 7f4e40600640 1 -- 172.21.15.102:0/908493730 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e381a0f60 con 0x7f4e38102980 2024-04-26T20:31:49.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.003+0000 7f4e36000640 1 -- 172.21.15.102:0/908493730 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24035030 con 0x7f4e38102980 2024-04-26T20:31:49.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.005+0000 7f4e40600640 1 -- 172.21.15.102:0/908493730 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e38109ca0 con 0x7f4e38102980 2024-04-26T20:31:49.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.006+0000 7f4e36000640 1 -- 172.21.15.102:0/908493730 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f4e24042070 con 0x7f4e38102980 2024-04-26T20:31:49.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.008+0000 7f4e36000640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4e200a3c30 0x7f4e200a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:49.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.008+0000 7f4e36000640 1 -- 172.21.15.102:0/908493730 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4e240f07b0 con 0x7f4e38102980 2024-04-26T20:31:49.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.008+0000 7f4e3d200640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4e200a3c30 0x7f4e200a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:49.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.009+0000 7f4e3d200640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4e200a3c30 0x7f4e200a60f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4e3819a080 tx=0x7f4e30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:49.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.012+0000 7f4e36000640 1 -- 172.21.15.102:0/908493730 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4e240bd590 con 0x7f4e38102980 2024-04-26T20:31:49.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.256+0000 7f4e40600640 1 -- 172.21.15.102:0/908493730 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e38063530 con 0x7f4e38102980 2024-04-26T20:31:49.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.257+0000 7f4e36000640 1 -- 172.21.15.102:0/908493730 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f4e24031e10 con 0x7f4e38102980 2024-04-26T20:31:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:49.259 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T20:31:49.259 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 3 2024-04-26T20:31:49.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 -- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4e200a3c30 msgr2=0x7f4e200a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:49.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4e200a3c30 0x7f4e200a60f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4e3819a080 tx=0x7f4e30005f90 comp rx=0 tx=0).stop 2024-04-26T20:31:49.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 -- 172.21.15.102:0/908493730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e38102980 msgr2=0x7f4e38198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:49.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e38102980 0x7f4e38198b70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e24005ab0 tx=0x7f4e24003c10 comp rx=0 tx=0).stop 2024-04-26T20:31:49.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 -- 172.21.15.102:0/908493730 shutdown_connections 2024-04-26T20:31:49.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4e200a3c30 0x7f4e200a60f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:49.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e38103b70 0x7f4e381990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:49.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e38104530 0x7f4e381a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:49.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 --2- 172.21.15.102:0/908493730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4e38102980 0x7f4e38198b70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:49.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 -- 172.21.15.102:0/908493730 >> 172.21.15.102:0/908493730 conn(0x7f4e380fdf10 msgr2=0x7f4e38106e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:49.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 -- 172.21.15.102:0/908493730 shutdown_connections 2024-04-26T20:31:49.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:49.260+0000 7f4e40600640 1 -- 172.21.15.102:0/908493730 wait complete. 2024-04-26T20:31:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:49 smithi130 ceph-mon[32704]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:49.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:49 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/908493730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:49.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:49 smithi102 ceph-mon[28094]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:49.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:49 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/908493730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:49.899 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:31:49 smithi102 podman[68107]: 2024-04-26 20:31:49.627703174 +0000 UTC m=+0.188983058 container create 0badd3389a910845252705c1113280712e069a0a1062347d9dda2d273c08324d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:31:50.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:49 smithi112 ceph-mon[32769]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:50.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:49 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/908493730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:50.399 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:31:49 smithi102 podman[68107]: 2024-04-26 20:31:49.919293634 +0000 UTC m=+0.480573528 container init 0badd3389a910845252705c1113280712e069a0a1062347d9dda2d273c08324d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T20:31:50.399 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:31:49 smithi102 podman[68107]: 2024-04-26 20:31:49.929228518 +0000 UTC m=+0.490508435 container start 0badd3389a910845252705c1113280712e069a0a1062347d9dda2d273c08324d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:31:50.399 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:31:49 smithi102 ceph-mds[68126]: -- 172.21.15.102:0/1106831040 <== mon.0 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555984f0da40 con 0x555984ee1000 2024-04-26T20:31:50.399 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:31:49 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-j[68122]: starting mds.j at 2024-04-26T20:31:50.399 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:31:50 smithi102 bash[68107]: 0badd3389a910845252705c1113280712e069a0a1062347d9dda2d273c08324d 2024-04-26T20:31:50.399 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:31:50 smithi102 systemd[1]: Started Ceph mds.j for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:31:51.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:31:51.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:51.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:51.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:51.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:51.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:51.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:51.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: Deploying daemon mds.d on smithi102 2024-04-26T20:31:51.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:51.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.102:6834/3853049115,v1:172.21.15.102:6835/3853049115] up:boot 2024-04-26T20:31:51.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: fsmap 3 up:standby 2024-04-26T20:31:51.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:51 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T20:31:51.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.490+0000 7f5356a00640 1 -- 172.21.15.102:0/392472801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5350106740 msgr2=0x7f535010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:51.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.490+0000 7f5356a00640 1 --2- 172.21.15.102:0/392472801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5350106740 0x7f535010cfd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5348009be0 tx=0x7f534802a6d0 comp rx=0 tx=0).stop 2024-04-26T20:31:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.490+0000 7f5356a00640 1 -- 172.21.15.102:0/392472801 shutdown_connections 2024-04-26T20:31:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.490+0000 7f5356a00640 1 --2- 172.21.15.102:0/392472801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5350105d80 0x7f5350106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.490+0000 7f5356a00640 1 --2- 172.21.15.102:0/392472801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5350106740 0x7f535010cfd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.490+0000 7f5356a00640 1 --2- 172.21.15.102:0/392472801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5350104b90 0x7f5350104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.490+0000 7f5356a00640 1 -- 172.21.15.102:0/392472801 >> 172.21.15.102:0/392472801 conn(0x7f5350100120 msgr2=0x7f5350102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.490+0000 7f5356a00640 1 -- 172.21.15.102:0/392472801 shutdown_connections 2024-04-26T20:31:51.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.490+0000 7f5356a00640 1 -- 172.21.15.102:0/392472801 wait complete. 2024-04-26T20:31:51.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.491+0000 7f5356a00640 1 Processor -- start 2024-04-26T20:31:51.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.491+0000 7f5356a00640 1 -- start start 2024-04-26T20:31:51.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f5356a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5350104b90 0x7f535019ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:51.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f5356a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5350105d80 0x7f535019b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f5356a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5350106740 0x7f53501a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f5356a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f53501106a0 con 0x7f5350104b90 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f5356a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5350110520 con 0x7f5350105d80 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f5356a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5350110820 con 0x7f5350106740 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f534fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5350104b90 0x7f535019ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f534f400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5350105d80 0x7f535019b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f534fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5350104b90 0x7f535019ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37794/0 (socket says 172.21.15.102:37794) 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f534f400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5350105d80 0x7f535019b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:48040/0 (socket says 172.21.15.102:48040) 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f534fe00640 1 -- 172.21.15.102:0/3166175910 learned_addr learned my addr 172.21.15.102:0/3166175910 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f5354a00640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5350106740 0x7f53501a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f534f400640 1 -- 172.21.15.102:0/3166175910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5350106740 msgr2=0x7f53501a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f534f400640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5350106740 0x7f53501a22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f534f400640 1 -- 172.21.15.102:0/3166175910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5350104b90 msgr2=0x7f535019ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:51.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f534f400640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5350104b90 0x7f535019ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.492+0000 7f534f400640 1 -- 172.21.15.102:0/3166175910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53501a2940 con 0x7f5350105d80 2024-04-26T20:31:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.493+0000 7f534f400640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5350105d80 0x7f535019b270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f533c00d6d0 tx=0x7f533c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.493+0000 7f534cc00640 1 -- 172.21.15.102:0/3166175910 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c017070 con 0x7f5350105d80 2024-04-26T20:31:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.493+0000 7f534cc00640 1 -- 172.21.15.102:0/3166175910 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f533c009820 con 0x7f5350105d80 2024-04-26T20:31:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.493+0000 7f534cc00640 1 -- 172.21.15.102:0/3166175910 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c004c40 con 0x7f5350105d80 2024-04-26T20:31:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.493+0000 7f5356a00640 1 -- 172.21.15.102:0/3166175910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53501a2c30 con 0x7f5350105d80 2024-04-26T20:31:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.493+0000 7f5356a00640 1 -- 172.21.15.102:0/3166175910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53501a3130 con 0x7f5350105d80 2024-04-26T20:31:51.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.494+0000 7f5356a00640 1 -- 172.21.15.102:0/3166175910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f535010beb0 con 0x7f5350105d80 2024-04-26T20:31:51.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.497+0000 7f534cc00640 1 -- 172.21.15.102:0/3166175910 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f533c005220 con 0x7f5350105d80 2024-04-26T20:31:51.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.498+0000 7f534cc00640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53340a3bb0 0x7f53340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:51.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.498+0000 7f534cc00640 1 -- 172.21.15.102:0/3166175910 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f533c0cfd00 con 0x7f5350105d80 2024-04-26T20:31:51.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.499+0000 7f534fe00640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53340a3bb0 0x7f53340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:51.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.499+0000 7f534fe00640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53340a3bb0 0x7f53340a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f535019c200 tx=0x7f5340009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:51.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.500+0000 7f534cc00640 1 -- 172.21.15.102:0/3166175910 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f533c09c490 con 0x7f5350105d80 2024-04-26T20:31:51.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:51.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:51.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:51.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:51.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:51.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:51.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: Deploying daemon mds.d on smithi102 2024-04-26T20:31:51.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:51.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.102:6834/3853049115,v1:172.21.15.102:6835/3853049115] up:boot 2024-04-26T20:31:51.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: fsmap 3 up:standby 2024-04-26T20:31:51.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:51 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T20:31:51.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: Deploying daemon mds.d on smithi102 2024-04-26T20:31:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.102:6834/3853049115,v1:172.21.15.102:6835/3853049115] up:boot 2024-04-26T20:31:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: fsmap 3 up:standby 2024-04-26T20:31:51.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:51 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T20:31:51.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.747+0000 7f5356a00640 1 -- 172.21.15.102:0/3166175910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f535002dc90 con 0x7f5350105d80 2024-04-26T20:31:51.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.748+0000 7f534cc00640 1 -- 172.21.15.102:0/3166175910 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3052 (secure 0 0 0) 0x7f533c0a0790 con 0x7f5350105d80 2024-04-26T20:31:51.749 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:51.749 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T20:31:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 4 2024-04-26T20:31:51.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.750+0000 7f5356a00640 1 -- 172.21.15.102:0/3166175910 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53340a3bb0 msgr2=0x7f53340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.750+0000 7f5356a00640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53340a3bb0 0x7f53340a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f535019c200 tx=0x7f5340009260 comp rx=0 tx=0).stop 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.750+0000 7f5356a00640 1 -- 172.21.15.102:0/3166175910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5350105d80 msgr2=0x7f535019b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.750+0000 7f5356a00640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5350105d80 0x7f535019b270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f533c00d6d0 tx=0x7f533c00db90 comp rx=0 tx=0).stop 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.750+0000 7f5356a00640 1 -- 172.21.15.102:0/3166175910 shutdown_connections 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.750+0000 7f5356a00640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53340a3bb0 0x7f53340a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.750+0000 7f5356a00640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5350106740 0x7f53501a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.750+0000 7f5356a00640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5350105d80 0x7f535019b270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.750+0000 7f5356a00640 1 --2- 172.21.15.102:0/3166175910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5350104b90 0x7f535019ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.750+0000 7f5356a00640 1 -- 172.21.15.102:0/3166175910 >> 172.21.15.102:0/3166175910 conn(0x7f5350100120 msgr2=0x7f5350109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.751+0000 7f5356a00640 1 -- 172.21.15.102:0/3166175910 shutdown_connections 2024-04-26T20:31:51.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:51.751+0000 7f5356a00640 1 -- 172.21.15.102:0/3166175910 wait complete. 2024-04-26T20:31:52.484 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:52 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3166175910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:52.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:52 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3166175910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:52.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:52 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3166175910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:53.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:53 smithi102 ceph-mon[28094]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:53.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:53 smithi112 ceph-mon[32769]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:53.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:53 smithi130 ceph-mon[32704]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:53.649 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:31:53 smithi102 systemd[1]: Starting Ceph mds.d for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:31:53.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:31:53.972 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:31:53 smithi102 podman[68453]: 2024-04-26 20:31:53.560717686 +0000 UTC m=+0.036621264 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:31:53.972 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:31:53 smithi102 podman[68453]: 2024-04-26 20:31:53.675994327 +0000 UTC m=+0.151897937 container create de3b9911ca8d39ca15e53834749d8ea9962b25aacf5b2d126f6b0f78318fbff3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T20:31:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.986+0000 7fee5a200640 1 -- 172.21.15.102:0/4283198109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee54102860 msgr2=0x7fee54102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.986+0000 7fee5a200640 1 --2- 172.21.15.102:0/4283198109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee54102860 0x7fee54102ce0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009900 tx=0x7fee3c02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:31:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.986+0000 7fee5a200640 1 -- 172.21.15.102:0/4283198109 shutdown_connections 2024-04-26T20:31:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.986+0000 7fee5a200640 1 --2- 172.21.15.102:0/4283198109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee54102860 0x7fee54102ce0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.986+0000 7fee5a200640 1 --2- 172.21.15.102:0/4283198109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee54103220 0x7fee54109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.986+0000 7fee5a200640 1 --2- 172.21.15.102:0/4283198109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee54101670 0x7fee54101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:53.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.986+0000 7fee5a200640 1 -- 172.21.15.102:0/4283198109 >> 172.21.15.102:0/4283198109 conn(0x7fee540fcbc0 msgr2=0x7fee540ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:53.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.986+0000 7fee5a200640 1 -- 172.21.15.102:0/4283198109 shutdown_connections 2024-04-26T20:31:53.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.986+0000 7fee5a200640 1 -- 172.21.15.102:0/4283198109 wait complete. 2024-04-26T20:31:53.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.987+0000 7fee5a200640 1 Processor -- start 2024-04-26T20:31:53.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.987+0000 7fee5a200640 1 -- start start 2024-04-26T20:31:53.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.987+0000 7fee5a200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee54101670 0x7fee5419abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:53.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.987+0000 7fee5a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee54102860 0x7fee5419b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:53.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.987+0000 7fee5a200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee54103220 0x7fee541a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.987+0000 7fee5a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fee54069e50 con 0x7fee54102860 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.987+0000 7fee5a200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fee54069cd0 con 0x7fee54103220 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.987+0000 7fee5a200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fee54069fd0 con 0x7fee54101670 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee4be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee54102860 0x7fee5419b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee58e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee54101670 0x7fee5419abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee58e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee54101670 0x7fee5419abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:58184/0 (socket says 172.21.15.102:58184) 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee4be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee54102860 0x7fee5419b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37802/0 (socket says 172.21.15.102:37802) 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee58e00640 1 -- 172.21.15.102:0/2625632801 learned_addr learned my addr 172.21.15.102:0/2625632801 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee59800640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee54103220 0x7fee541a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee58e00640 1 -- 172.21.15.102:0/2625632801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee54103220 msgr2=0x7fee541a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee58e00640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee54103220 0x7fee541a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee58e00640 1 -- 172.21.15.102:0/2625632801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee54102860 msgr2=0x7fee5419b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee58e00640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee54102860 0x7fee5419b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.988+0000 7fee58e00640 1 -- 172.21.15.102:0/2625632801 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee541a27e0 con 0x7fee54101670 2024-04-26T20:31:53.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.989+0000 7fee58e00640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee54101670 0x7fee5419abc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee44009d20 tx=0x7fee4400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:53.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.989+0000 7fee52000640 1 -- 172.21.15.102:0/2625632801 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee44010630 con 0x7fee54101670 2024-04-26T20:31:53.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.989+0000 7fee5a200640 1 -- 172.21.15.102:0/2625632801 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee541a2ad0 con 0x7fee54101670 2024-04-26T20:31:53.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.989+0000 7fee52000640 1 -- 172.21.15.102:0/2625632801 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee44010c80 con 0x7fee54101670 2024-04-26T20:31:53.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.989+0000 7fee52000640 1 -- 172.21.15.102:0/2625632801 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee44018d50 con 0x7fee54101670 2024-04-26T20:31:53.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.989+0000 7fee5a200640 1 -- 172.21.15.102:0/2625632801 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee541a3040 con 0x7fee54101670 2024-04-26T20:31:53.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.991+0000 7fee5a200640 1 -- 172.21.15.102:0/2625632801 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee541a3210 con 0x7fee54101670 2024-04-26T20:31:53.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.992+0000 7fee52000640 1 -- 172.21.15.102:0/2625632801 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fee4401f020 con 0x7fee54101670 2024-04-26T20:31:53.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.994+0000 7fee52000640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3c30 0x7fee380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.994+0000 7fee52000640 1 -- 172.21.15.102:0/2625632801 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fee440d03a0 con 0x7fee54101670 2024-04-26T20:31:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.994+0000 7fee4be00640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3c30 0x7fee380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.995+0000 7fee4be00640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3c30 0x7fee380a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000c00 tx=0x7fee3c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:53.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:53.998+0000 7fee52000640 1 -- 172.21.15.102:0/2625632801 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fee4409d180 con 0x7fee54101670 2024-04-26T20:31:54.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.255+0000 7fee5a200640 1 -- 172.21.15.102:0/2625632801 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee541a33e0 con 0x7fee54101670 2024-04-26T20:31:54.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.256+0000 7fee52000640 1 -- 172.21.15.102:0/2625632801 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3880 (secure 0 0 0) 0x7fee4409ca80 con 0x7fee54101670 2024-04-26T20:31:54.258 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:54.258 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-26T20:31:54.258 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 5 2024-04-26T20:31:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.259+0000 7fee5a200640 1 -- 172.21.15.102:0/2625632801 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3c30 msgr2=0x7fee380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.259+0000 7fee5a200640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3c30 0x7fee380a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000c00 tx=0x7fee3c0314a0 comp rx=0 tx=0).stop 2024-04-26T20:31:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.259+0000 7fee5a200640 1 -- 172.21.15.102:0/2625632801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee54101670 msgr2=0x7fee5419abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.259+0000 7fee5a200640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee54101670 0x7fee5419abc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee44009d20 tx=0x7fee4400c6a0 comp rx=0 tx=0).stop 2024-04-26T20:31:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.260+0000 7fee5a200640 1 -- 172.21.15.102:0/2625632801 shutdown_connections 2024-04-26T20:31:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.260+0000 7fee5a200640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3c30 0x7fee380a60f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.260+0000 7fee5a200640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee54102860 0x7fee5419b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.260+0000 7fee5a200640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee54103220 0x7fee541a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.260+0000 7fee5a200640 1 --2- 172.21.15.102:0/2625632801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee54101670 0x7fee5419abc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.260+0000 7fee5a200640 1 -- 172.21.15.102:0/2625632801 >> 172.21.15.102:0/2625632801 conn(0x7fee540fcbc0 msgr2=0x7fee540fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.260+0000 7fee5a200640 1 -- 172.21.15.102:0/2625632801 shutdown_connections 2024-04-26T20:31:54.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:54.260+0000 7fee5a200640 1 -- 172.21.15.102:0/2625632801 wait complete. 2024-04-26T20:31:54.277 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:31:53 smithi102 podman[68453]: 2024-04-26 20:31:53.971394174 +0000 UTC m=+0.447297791 container init de3b9911ca8d39ca15e53834749d8ea9962b25aacf5b2d126f6b0f78318fbff3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T20:31:54.277 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:31:53 smithi102 podman[68453]: 2024-04-26 20:31:53.97969293 +0000 UTC m=+0.455596517 container start de3b9911ca8d39ca15e53834749d8ea9962b25aacf5b2d126f6b0f78318fbff3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:31:54.277 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:31:54 smithi102 ceph-mds[68494]: -- 172.21.15.102:0/1503325143 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c88182da40 con 0x55c881801000 2024-04-26T20:31:54.277 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:31:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d[68484]: starting mds.d at 2024-04-26T20:31:54.277 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:31:54 smithi102 bash[68453]: de3b9911ca8d39ca15e53834749d8ea9962b25aacf5b2d126f6b0f78318fbff3 2024-04-26T20:31:54.277 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:31:54 smithi102 systemd[1]: Started Ceph mds.d for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:31:55.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:55.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:55.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:55.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:55.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:55.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:55.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: Deploying daemon mds.e on smithi112 2024-04-26T20:31:55.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.102:6836/3283929332,v1:172.21.15.102:6837/3283929332] up:boot 2024-04-26T20:31:55.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: fsmap 4 up:standby 2024-04-26T20:31:55.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T20:31:55.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2625632801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:55.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:55 smithi112 ceph-mon[32769]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: Deploying daemon mds.e on smithi112 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.102:6836/3283929332,v1:172.21.15.102:6837/3283929332] up:boot 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: fsmap 4 up:standby 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T20:31:55.552 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2625632801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:55.553 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:55 smithi102 ceph-mon[28094]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: Deploying daemon mds.e on smithi112 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.102:6836/3283929332,v1:172.21.15.102:6837/3283929332] up:boot 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: fsmap 4 up:standby 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2625632801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:55.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:55 smithi130 ceph-mon[32704]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:56.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:31:56.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.486+0000 7f7d1ce00640 1 -- 172.21.15.102:0/1209273324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d18103b70 msgr2=0x7f7d18103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:56.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.486+0000 7f7d1ce00640 1 --2- 172.21.15.102:0/1209273324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d18103b70 0x7f7d18103ff0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009900 tx=0x7f7d0402a410 comp rx=0 tx=0).stop 2024-04-26T20:31:56.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.486+0000 7f7d1ce00640 1 -- 172.21.15.102:0/1209273324 shutdown_connections 2024-04-26T20:31:56.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.486+0000 7f7d1ce00640 1 --2- 172.21.15.102:0/1209273324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d18103b70 0x7f7d18103ff0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:56.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.486+0000 7f7d1ce00640 1 --2- 172.21.15.102:0/1209273324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d18104530 0x7f7d1810adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:56.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.486+0000 7f7d1ce00640 1 --2- 172.21.15.102:0/1209273324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d18102980 0x7f7d18102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:56.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.486+0000 7f7d1ce00640 1 -- 172.21.15.102:0/1209273324 >> 172.21.15.102:0/1209273324 conn(0x7f7d180fdf10 msgr2=0x7f7d18100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:56.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.486+0000 7f7d1ce00640 1 -- 172.21.15.102:0/1209273324 shutdown_connections 2024-04-26T20:31:56.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.486+0000 7f7d1ce00640 1 -- 172.21.15.102:0/1209273324 wait complete. 2024-04-26T20:31:56.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.487+0000 7f7d1ce00640 1 Processor -- start 2024-04-26T20:31:56.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.487+0000 7f7d1ce00640 1 -- start start 2024-04-26T20:31:56.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.487+0000 7f7d1ce00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d18102980 0x7f7d18198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:56.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.487+0000 7f7d1ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d18103b70 0x7f7d181990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:56.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.487+0000 7f7d1ce00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d18104530 0x7f7d181a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:56.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d1ce00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1810e500 con 0x7f7d18103b70 2024-04-26T20:31:56.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d1ce00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1810e380 con 0x7f7d18104530 2024-04-26T20:31:56.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d1ce00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1810e680 con 0x7f7d18102980 2024-04-26T20:31:56.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d16800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d18104530 0x7f7d181a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d16800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d18104530 0x7f7d181a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:36226/0 (socket says 172.21.15.102:36226) 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d16800640 1 -- 172.21.15.102:0/2344339542 learned_addr learned my addr 172.21.15.102:0/2344339542 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d15e00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d18102980 0x7f7d18198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d15400640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d18103b70 0x7f7d181990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d16800640 1 -- 172.21.15.102:0/2344339542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d18102980 msgr2=0x7f7d18198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d16800640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d18102980 0x7f7d18198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d16800640 1 -- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d18103b70 msgr2=0x7f7d181990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d16800640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d18103b70 0x7f7d181990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d16800640 1 -- 172.21.15.102:0/2344339542 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d181a07d0 con 0x7f7d18104530 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d15400640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d18103b70 0x7f7d181990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.488+0000 7f7d15e00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d18102980 0x7f7d18198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.489+0000 7f7d16800640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d18104530 0x7f7d181a0160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000eb10 tx=0x7f7d1000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.489+0000 7f7d0ea00640 1 -- 172.21.15.102:0/2344339542 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10004b20 con 0x7f7d18104530 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.489+0000 7f7d1ce00640 1 -- 172.21.15.102:0/2344339542 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d181a0ac0 con 0x7f7d18104530 2024-04-26T20:31:56.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.489+0000 7f7d0ea00640 1 -- 172.21.15.102:0/2344339542 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d10004cc0 con 0x7f7d18104530 2024-04-26T20:31:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.489+0000 7f7d0ea00640 1 -- 172.21.15.102:0/2344339542 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10010670 con 0x7f7d18104530 2024-04-26T20:31:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.489+0000 7f7d1ce00640 1 -- 172.21.15.102:0/2344339542 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d181a1010 con 0x7f7d18104530 2024-04-26T20:31:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.490+0000 7f7d1ce00640 1 -- 172.21.15.102:0/2344339542 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d18109d70 con 0x7f7d18104530 2024-04-26T20:31:56.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.494+0000 7f7d0ea00640 1 -- 172.21.15.102:0/2344339542 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f7d10005020 con 0x7f7d18104530 2024-04-26T20:31:56.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.495+0000 7f7d0ea00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7cfc0a3bb0 0x7f7cfc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:56.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.495+0000 7f7d0ea00640 1 -- 172.21.15.102:0/2344339542 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7d100d05d0 con 0x7f7d18104530 2024-04-26T20:31:56.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.495+0000 7f7d15e00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7cfc0a3bb0 0x7f7cfc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:56.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.496+0000 7f7d15e00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7cfc0a3bb0 0x7f7cfc0a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d181039d0 tx=0x7f7d08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:56.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.499+0000 7f7d0ea00640 1 -- 172.21.15.102:0/2344339542 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7d1009d3b0 con 0x7f7d18104530 2024-04-26T20:31:56.703 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:31:56 smithi112 systemd[1]: Starting Ceph mds.e for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:31:56.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.748+0000 7f7d1ce00640 1 -- 172.21.15.102:0/2344339542 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d180635d0 con 0x7f7d18104530 2024-04-26T20:31:56.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.748+0000 7f7d0ea00640 1 -- 172.21.15.102:0/2344339542 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3880 (secure 0 0 0) 0x7f7d1009ccb0 con 0x7f7d18104530 2024-04-26T20:31:56.750 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:56.750 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-26T20:31:56.750 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 5 2024-04-26T20:31:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 -- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7cfc0a3bb0 msgr2=0x7f7cfc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7cfc0a3bb0 0x7f7cfc0a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d181039d0 tx=0x7f7d08005f90 comp rx=0 tx=0).stop 2024-04-26T20:31:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 -- 172.21.15.102:0/2344339542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d18104530 msgr2=0x7f7d181a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d18104530 0x7f7d181a0160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000eb10 tx=0x7f7d1000c2d0 comp rx=0 tx=0).stop 2024-04-26T20:31:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 -- 172.21.15.102:0/2344339542 shutdown_connections 2024-04-26T20:31:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7cfc0a3bb0 0x7f7cfc0a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d18103b70 0x7f7d181990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:56.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d18104530 0x7f7d181a0160 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:56.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 --2- 172.21.15.102:0/2344339542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d18102980 0x7f7d18198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:56.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 -- 172.21.15.102:0/2344339542 >> 172.21.15.102:0/2344339542 conn(0x7f7d180fdf10 msgr2=0x7f7d18106f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:56.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 -- 172.21.15.102:0/2344339542 shutdown_connections 2024-04-26T20:31:56.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:56.751+0000 7f7d1ce00640 1 -- 172.21.15.102:0/2344339542 wait complete. 2024-04-26T20:31:57.026 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:31:56 smithi112 podman[62203]: 2024-04-26 20:31:56.606073881 +0000 UTC m=+0.018298495 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:31:57.026 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:31:56 smithi112 podman[62203]: 2024-04-26 20:31:56.744762955 +0000 UTC m=+0.156987572 container create 29e7af1817b9d3a98bb600c6802ff9062b3a1cedb01b6d0dc538c8918edc37e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T20:31:57.026 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:31:57 smithi112 podman[62203]: 2024-04-26 20:31:57.026172885 +0000 UTC m=+0.438397499 container init 29e7af1817b9d3a98bb600c6802ff9062b3a1cedb01b6d0dc538c8918edc37e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:31:57.276 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:31:57 smithi112 podman[62203]: 2024-04-26 20:31:57.028817598 +0000 UTC m=+0.441042220 container start 29e7af1817b9d3a98bb600c6802ff9062b3a1cedb01b6d0dc538c8918edc37e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-26T20:31:57.277 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:31:57 smithi112 ceph-mds[62222]: -- 172.21.15.112:0/3180267517 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d08a9d5a40 con 0x55d08a9a8800 2024-04-26T20:31:57.277 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:31:57 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e[62218]: starting mds.e at 2024-04-26T20:31:57.277 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:31:57 smithi112 bash[62203]: 29e7af1817b9d3a98bb600c6802ff9062b3a1cedb01b6d0dc538c8918edc37e3 2024-04-26T20:31:57.277 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:31:57 smithi112 systemd[1]: Started Ceph mds.e for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:31:57.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:57 smithi130 ceph-mon[32704]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:57.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:57 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2344339542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:57.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:57 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:57.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:57 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:57.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:57 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:57.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:57 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:57.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:57 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:57.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:57 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:57 smithi102 ceph-mon[28094]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:57 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2344339542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:57 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:57.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:57 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:57.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:57 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:57.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:57 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:57.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:57 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:57.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:57 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:57.915 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:57 smithi112 ceph-mon[32769]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:57.916 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:57 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2344339542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:57.916 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:57 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:57.916 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:57 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:57.916 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:57 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:31:57.916 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:57 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:31:57.916 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:57 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:31:57.916 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:57 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:31:58.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:31:58.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:58 smithi112 ceph-mon[32769]: Deploying daemon mds.f on smithi130 2024-04-26T20:31:58.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:58 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.112:6834/2010209065,v1:172.21.15.112:6835/2010209065] up:boot 2024-04-26T20:31:58.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:58 smithi112 ceph-mon[32769]: fsmap 5 up:standby 2024-04-26T20:31:58.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:58 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T20:31:58.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:58 smithi102 ceph-mon[28094]: Deploying daemon mds.f on smithi130 2024-04-26T20:31:58.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:58 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.112:6834/2010209065,v1:172.21.15.112:6835/2010209065] up:boot 2024-04-26T20:31:58.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:58 smithi102 ceph-mon[28094]: fsmap 5 up:standby 2024-04-26T20:31:58.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:58 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T20:31:58.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:58 smithi130 ceph-mon[32704]: Deploying daemon mds.f on smithi130 2024-04-26T20:31:58.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:58 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.112:6834/2010209065,v1:172.21.15.112:6835/2010209065] up:boot 2024-04-26T20:31:58.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:58 smithi130 ceph-mon[32704]: fsmap 5 up:standby 2024-04-26T20:31:58.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:58 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T20:31:58.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.983+0000 7f5904a00640 1 -- 172.21.15.102:0/3160465237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5900108d90 msgr2=0x7f59001091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:58.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.983+0000 7f5904a00640 1 --2- 172.21.15.102:0/3160465237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5900108d90 0x7f59001091b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009ca0 tx=0x7f58f002a8c0 comp rx=0 tx=0).stop 2024-04-26T20:31:58.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.983+0000 7f5904a00640 1 -- 172.21.15.102:0/3160465237 shutdown_connections 2024-04-26T20:31:58.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.983+0000 7f5904a00640 1 --2- 172.21.15.102:0/3160465237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f590010a940 0x7f59001111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:58.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.983+0000 7f5904a00640 1 --2- 172.21.15.102:0/3160465237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5900109f80 0x7f590010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:58.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.983+0000 7f5904a00640 1 --2- 172.21.15.102:0/3160465237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5900108d90 0x7f59001091b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:58.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.983+0000 7f5904a00640 1 -- 172.21.15.102:0/3160465237 >> 172.21.15.102:0/3160465237 conn(0x7f5900077d30 msgr2=0x7f590007a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:58.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.983+0000 7f5904a00640 1 -- 172.21.15.102:0/3160465237 shutdown_connections 2024-04-26T20:31:58.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.983+0000 7f5904a00640 1 -- 172.21.15.102:0/3160465237 wait complete. 2024-04-26T20:31:58.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.984+0000 7f5904a00640 1 Processor -- start 2024-04-26T20:31:58.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.984+0000 7f5904a00640 1 -- start start 2024-04-26T20:31:58.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.984+0000 7f5904a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5900108d90 0x7f590019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:58.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.984+0000 7f5904a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5900109f80 0x7f590019f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:58.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.984+0000 7f5904a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f590010a940 0x7f59001a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:58.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.984+0000 7f5904a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5900114890 con 0x7f5900109f80 2024-04-26T20:31:58.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.984+0000 7f5904a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5900114710 con 0x7f590010a940 2024-04-26T20:31:58.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.984+0000 7f5904a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5900114a10 con 0x7f5900108d90 2024-04-26T20:31:58.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58f5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5900109f80 0x7f590019f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58f5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5900109f80 0x7f590019f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44314/0 (socket says 172.21.15.102:44314) 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58f5e00640 1 -- 172.21.15.102:0/3273443688 learned_addr learned my addr 172.21.15.102:0/3273443688 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58ff200640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f590010a940 0x7f59001a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58fe800640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5900108d90 0x7f590019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58f5e00640 1 -- 172.21.15.102:0/3273443688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5900108d90 msgr2=0x7f590019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58f5e00640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5900108d90 0x7f590019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58f5e00640 1 -- 172.21.15.102:0/3273443688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f590010a940 msgr2=0x7f59001a6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58f5e00640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f590010a940 0x7f59001a6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58f5e00640 1 -- 172.21.15.102:0/3273443688 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59001a6bc0 con 0x7f5900109f80 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58ff200640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f590010a940 0x7f59001a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.985+0000 7f58f5e00640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5900109f80 0x7f590019f580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f58e800ea60 tx=0x7f58e800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.986+0000 7f58f7e00640 1 -- 172.21.15.102:0/3273443688 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e800cde0 con 0x7f5900109f80 2024-04-26T20:31:58.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.986+0000 7f58f7e00640 1 -- 172.21.15.102:0/3273443688 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58e80035f0 con 0x7f5900109f80 2024-04-26T20:31:58.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.986+0000 7f5904a00640 1 -- 172.21.15.102:0/3273443688 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59001a6eb0 con 0x7f5900109f80 2024-04-26T20:31:58.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.986+0000 7f58f7e00640 1 -- 172.21.15.102:0/3273443688 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e8010690 con 0x7f5900109f80 2024-04-26T20:31:58.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.986+0000 7f5904a00640 1 -- 172.21.15.102:0/3273443688 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59001a7370 con 0x7f5900109f80 2024-04-26T20:31:58.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.987+0000 7f5904a00640 1 -- 172.21.15.102:0/3273443688 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5900110180 con 0x7f5900109f80 2024-04-26T20:31:58.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.989+0000 7f58f7e00640 1 -- 172.21.15.102:0/3273443688 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f58e80037a0 con 0x7f5900109f80 2024-04-26T20:31:58.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.990+0000 7f58f7e00640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f58e40a3bb0 0x7f58e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:31:58.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.990+0000 7f58f7e00640 1 -- 172.21.15.102:0/3273443688 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f58e80d02a0 con 0x7f5900109f80 2024-04-26T20:31:58.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.990+0000 7f58fe800640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f58e40a3bb0 0x7f58e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:31:58.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.991+0000 7f58fe800640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f58e40a3bb0 0x7f58e40a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f58f0000f80 tx=0x7f58f0005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:31:58.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:58.995+0000 7f58f7e00640 1 -- 172.21.15.102:0/3273443688 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f58e809d080 con 0x7f5900109f80 2024-04-26T20:31:59.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.217+0000 7f5904a00640 1 -- 172.21.15.102:0/3273443688 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f590002dad0 con 0x7f5900109f80 2024-04-26T20:31:59.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.218+0000 7f58f7e00640 1 -- 172.21.15.102:0/3273443688 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4708 (secure 0 0 0) 0x7f58e809c980 con 0x7f5900109f80 2024-04-26T20:31:59.219 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:31:59.219 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T20:31:59.220 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 6 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.220+0000 7f5904a00640 1 -- 172.21.15.102:0/3273443688 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f58e40a3bb0 msgr2=0x7f58e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.220+0000 7f5904a00640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f58e40a3bb0 0x7f58e40a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f58f0000f80 tx=0x7f58f0005dc0 comp rx=0 tx=0).stop 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.221+0000 7f5904a00640 1 -- 172.21.15.102:0/3273443688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5900109f80 msgr2=0x7f590019f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.221+0000 7f5904a00640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5900109f80 0x7f590019f580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f58e800ea60 tx=0x7f58e800ef20 comp rx=0 tx=0).stop 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.221+0000 7f5904a00640 1 -- 172.21.15.102:0/3273443688 shutdown_connections 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.221+0000 7f5904a00640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f58e40a3bb0 0x7f58e40a6070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.221+0000 7f5904a00640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5900109f80 0x7f590019f580 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.221+0000 7f5904a00640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f590010a940 0x7f59001a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.221+0000 7f5904a00640 1 --2- 172.21.15.102:0/3273443688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5900108d90 0x7f590019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.221+0000 7f5904a00640 1 -- 172.21.15.102:0/3273443688 >> 172.21.15.102:0/3273443688 conn(0x7f5900077d30 msgr2=0x7f590010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:31:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.221+0000 7f5904a00640 1 -- 172.21.15.102:0/3273443688 shutdown_connections 2024-04-26T20:31:59.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:31:59.221+0000 7f5904a00640 1 -- 172.21.15.102:0/3273443688 wait complete. 2024-04-26T20:31:59.581 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:31:59 smithi130 systemd[1]: Starting Ceph mds.f for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:31:59.849 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:59 smithi130 ceph-mon[32704]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:59.850 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:31:59 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3273443688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:31:59.850 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:31:59 smithi130 podman[63021]: 2024-04-26 20:31:59.598195757 +0000 UTC m=+0.018775232 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:31:59.850 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:31:59 smithi130 podman[63021]: 2024-04-26 20:31:59.720668677 +0000 UTC m=+0.141248193 container create d3e425d7e51f3b00ce6e139ba53bea43a344540dc669783446d023a4df3b7599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-26T20:31:59.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:59 smithi102 ceph-mon[28094]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:31:59.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:31:59 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3273443688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:00.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:59 smithi112 ceph-mon[32769]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:00.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:31:59 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3273443688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:00.122 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:31:59 smithi130 podman[63021]: 2024-04-26 20:31:59.990389777 +0000 UTC m=+0.410969224 container init d3e425d7e51f3b00ce6e139ba53bea43a344540dc669783446d023a4df3b7599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-26T20:32:00.122 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:31:59 smithi130 podman[63021]: 2024-04-26 20:31:59.993124343 +0000 UTC m=+0.413703795 container start d3e425d7e51f3b00ce6e139ba53bea43a344540dc669783446d023a4df3b7599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T20:32:00.122 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:32:00 smithi130 ceph-mds[63041]: -- 172.21.15.130:0/1591296429 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56424ee67a40 con 0x56424ee3ac00 2024-04-26T20:32:00.122 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:32:00 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f[63037]: starting mds.f at 2024-04-26T20:32:00.122 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:32:00 smithi130 bash[63021]: d3e425d7e51f3b00ce6e139ba53bea43a344540dc669783446d023a4df3b7599 2024-04-26T20:32:00.122 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:32:00 smithi130 systemd[1]: Started Ceph mds.f for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:32:01.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:01.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: Deploying daemon mds.c on smithi130 2024-04-26T20:32:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.130:6834/829567775,v1:172.21.15.130:6835/829567775] up:boot 2024-04-26T20:32:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: fsmap 6 up:standby 2024-04-26T20:32:01.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:01 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: Deploying daemon mds.c on smithi130 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.130:6834/829567775,v1:172.21.15.130:6835/829567775] up:boot 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: fsmap 6 up:standby 2024-04-26T20:32:01.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:01 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:32:01.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.449+0000 7fc8d6600640 1 -- 172.21.15.102:0/962337739 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d00772f0 msgr2=0x7fc8d0077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:01.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.449+0000 7fc8d6600640 1 --2- 172.21.15.102:0/962337739 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d00772f0 0x7fc8d0077710 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009900 tx=0x7fc8c402a410 comp rx=0 tx=0).stop 2024-04-26T20:32:01.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.449+0000 7fc8d6600640 1 -- 172.21.15.102:0/962337739 shutdown_connections 2024-04-26T20:32:01.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.449+0000 7fc8d6600640 1 --2- 172.21.15.102:0/962337739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8d0078ed0 0x7fc8d0079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:01.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.449+0000 7fc8d6600640 1 --2- 172.21.15.102:0/962337739 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d0078530 0x7fc8d0078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:01.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.449+0000 7fc8d6600640 1 --2- 172.21.15.102:0/962337739 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d00772f0 0x7fc8d0077710 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:01.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.449+0000 7fc8d6600640 1 -- 172.21.15.102:0/962337739 >> 172.21.15.102:0/962337739 conn(0x7fc8d00fff10 msgr2=0x7fc8d0102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:01.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.449+0000 7fc8d6600640 1 -- 172.21.15.102:0/962337739 shutdown_connections 2024-04-26T20:32:01.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.449+0000 7fc8d6600640 1 -- 172.21.15.102:0/962337739 wait complete. 2024-04-26T20:32:01.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.450+0000 7fc8d6600640 1 Processor -- start 2024-04-26T20:32:01.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8d6600640 1 -- start start 2024-04-26T20:32:01.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8d6600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d00772f0 0x7fc8d019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:01.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8d6600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8d0078530 0x7fc8d019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:01.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8d6600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d0078ed0 0x7fc8d01a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8d6600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d01148c0 con 0x7fc8d0078ed0 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8d6600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0114740 con 0x7fc8d00772f0 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8d6600640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0114a40 con 0x7fc8d0078530 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8d5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d0078ed0 0x7fc8d01a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8cfe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8d0078530 0x7fc8d019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8cfe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8d0078530 0x7fc8d019f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:50194/0 (socket says 172.21.15.102:50194) 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8cfe00640 1 -- 172.21.15.102:0/1678218897 learned_addr learned my addr 172.21.15.102:0/1678218897 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.451+0000 7fc8d5200640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d00772f0 0x7fc8d019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8cfe00640 1 -- 172.21.15.102:0/1678218897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d00772f0 msgr2=0x7fc8d019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8cfe00640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d00772f0 0x7fc8d019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8cfe00640 1 -- 172.21.15.102:0/1678218897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d0078ed0 msgr2=0x7fc8d01a6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8cfe00640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d0078ed0 0x7fc8d01a6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:01.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8cfe00640 1 -- 172.21.15.102:0/1678218897 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d01a6b90 con 0x7fc8d0078530 2024-04-26T20:32:01.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8d5c00640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d0078ed0 0x7fc8d01a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:01.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8cfe00640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8d0078530 0x7fc8d019f4b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00ea60 tx=0x7fc8bc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:01.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8cd600640 1 -- 172.21.15.102:0/1678218897 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc00cde0 con 0x7fc8d0078530 2024-04-26T20:32:01.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8cd600640 1 -- 172.21.15.102:0/1678218897 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8bc0035f0 con 0x7fc8d0078530 2024-04-26T20:32:01.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8d6600640 1 -- 172.21.15.102:0/1678218897 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d01a6e80 con 0x7fc8d0078530 2024-04-26T20:32:01.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8cd600640 1 -- 172.21.15.102:0/1678218897 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc010690 con 0x7fc8d0078530 2024-04-26T20:32:01.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.452+0000 7fc8d6600640 1 -- 172.21.15.102:0/1678218897 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d01a7390 con 0x7fc8d0078530 2024-04-26T20:32:01.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.453+0000 7fc8d6600640 1 -- 172.21.15.102:0/1678218897 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d01112f0 con 0x7fc8d0078530 2024-04-26T20:32:01.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.455+0000 7fc8cd600640 1 -- 172.21.15.102:0/1678218897 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fc8bc0037a0 con 0x7fc8d0078530 2024-04-26T20:32:01.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.457+0000 7fc8cd600640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc8b80a3c30 0x7fc8b80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:01.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.457+0000 7fc8cd600640 1 -- 172.21.15.102:0/1678218897 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc8bc0d0230 con 0x7fc8d0078530 2024-04-26T20:32:01.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.457+0000 7fc8d5200640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc8b80a3c30 0x7fc8b80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:01.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.458+0000 7fc8d5200640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc8b80a3c30 0x7fc8b80a60f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4000c00 tx=0x7fc8c40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:01.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.461+0000 7fc8cd600640 1 -- 172.21.15.102:0/1678218897 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc8bc09d010 con 0x7fc8d0078530 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: Deploying daemon mds.c on smithi130 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.130:6834/829567775,v1:172.21.15.130:6835/829567775] up:boot 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: fsmap 6 up:standby 2024-04-26T20:32:01.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:01 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:32:01.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.721+0000 7fc8d6600640 1 -- 172.21.15.102:0/1678218897 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8d002dad0 con 0x7fc8d0078530 2024-04-26T20:32:01.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.722+0000 7fc8cd600640 1 -- 172.21.15.102:0/1678218897 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5533 (secure 0 0 0) 0x7fc8bc09c910 con 0x7fc8d0078530 2024-04-26T20:32:01.723 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:01.723 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-26T20:32:01.723 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 7 2024-04-26T20:32:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.724+0000 7fc8d6600640 1 -- 172.21.15.102:0/1678218897 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc8b80a3c30 msgr2=0x7fc8b80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.724+0000 7fc8d6600640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc8b80a3c30 0x7fc8b80a60f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4000c00 tx=0x7fc8c40314a0 comp rx=0 tx=0).stop 2024-04-26T20:32:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.724+0000 7fc8d6600640 1 -- 172.21.15.102:0/1678218897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8d0078530 msgr2=0x7fc8d019f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.724+0000 7fc8d6600640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8d0078530 0x7fc8d019f4b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc00ea60 tx=0x7fc8bc00ef20 comp rx=0 tx=0).stop 2024-04-26T20:32:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.724+0000 7fc8d6600640 1 -- 172.21.15.102:0/1678218897 shutdown_connections 2024-04-26T20:32:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.724+0000 7fc8d6600640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc8b80a3c30 0x7fc8b80a60f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.724+0000 7fc8d6600640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d0078ed0 0x7fc8d01a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.724+0000 7fc8d6600640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc8d0078530 0x7fc8d019f4b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.724+0000 7fc8d6600640 1 --2- 172.21.15.102:0/1678218897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d00772f0 0x7fc8d019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:01.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.724+0000 7fc8d6600640 1 -- 172.21.15.102:0/1678218897 >> 172.21.15.102:0/1678218897 conn(0x7fc8d00fff10 msgr2=0x7fc8d0101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.725+0000 7fc8d6600640 1 -- 172.21.15.102:0/1678218897 shutdown_connections 2024-04-26T20:32:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:01.725+0000 7fc8d6600640 1 -- 172.21.15.102:0/1678218897 wait complete. 2024-04-26T20:32:02.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:02 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1678218897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:02.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:02 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1678218897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:02.622 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:32:02 smithi130 systemd[1]: Starting Ceph mds.c for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:32:02.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:02 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1678218897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:02.883 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:32:02 smithi130 podman[63336]: 2024-04-26 20:32:02.53318562 +0000 UTC m=+0.017938693 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:32:02.883 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:32:02 smithi130 podman[63336]: 2024-04-26 20:32:02.654036586 +0000 UTC m=+0.138789685 container create dd597cbb22147206c39303027f1c061341cafc8c1b598ef26405edc131b2f93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:32:02.883 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:32:02 smithi130 podman[63336]: 2024-04-26 20:32:02.880179148 +0000 UTC m=+0.364932206 container init dd597cbb22147206c39303027f1c061341cafc8c1b598ef26405edc131b2f93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-26T20:32:02.883 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:32:02 smithi130 podman[63336]: 2024-04-26 20:32:02.882760609 +0000 UTC m=+0.367513669 container start dd597cbb22147206c39303027f1c061341cafc8c1b598ef26405edc131b2f93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-26T20:32:03.218 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:32:02 smithi130 ceph-mds[63355]: -- 172.21.15.130:0/207001313 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b6b487a40 con 0x559b6b45ac00 2024-04-26T20:32:03.218 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:32:02 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c[63351]: starting mds.c at 2024-04-26T20:32:03.218 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:32:02 smithi130 bash[63336]: dd597cbb22147206c39303027f1c061341cafc8c1b598ef26405edc131b2f93a 2024-04-26T20:32:03.218 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:32:02 smithi130 systemd[1]: Started Ceph mds.c for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:32:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:03 smithi112 ceph-mon[32769]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:03 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:03 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:03 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:03 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:03 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:03.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:03 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:03.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:03 smithi130 ceph-mon[32704]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:03.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:03 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:03.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:03 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:03.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:03 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:03.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:03 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:03.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:03 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:03.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:03 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:03.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:03 smithi102 ceph-mon[28094]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:03.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:03.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:03.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:03.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:03.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:03.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:03 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:03.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:03.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.960+0000 7ff2e6c00640 1 -- 172.21.15.102:0/2363617224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2e0109f80 msgr2=0x7ff2e010a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:03.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.960+0000 7ff2e6c00640 1 --2- 172.21.15.102:0/2363617224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2e0109f80 0x7ff2e010a400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009900 tx=0x7ff2cc02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:03.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.960+0000 7ff2e6c00640 1 -- 172.21.15.102:0/2363617224 shutdown_connections 2024-04-26T20:32:03.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.960+0000 7ff2e6c00640 1 --2- 172.21.15.102:0/2363617224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e010a940 0x7ff2e01111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:03.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.960+0000 7ff2e6c00640 1 --2- 172.21.15.102:0/2363617224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2e0109f80 0x7ff2e010a400 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:03.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.960+0000 7ff2e6c00640 1 --2- 172.21.15.102:0/2363617224 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2e0108d90 0x7ff2e01091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:03.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.960+0000 7ff2e6c00640 1 -- 172.21.15.102:0/2363617224 >> 172.21.15.102:0/2363617224 conn(0x7ff2e0077d30 msgr2=0x7ff2e007a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:03.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.960+0000 7ff2e6c00640 1 -- 172.21.15.102:0/2363617224 shutdown_connections 2024-04-26T20:32:03.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.960+0000 7ff2e6c00640 1 -- 172.21.15.102:0/2363617224 wait complete. 2024-04-26T20:32:03.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.961+0000 7ff2e6c00640 1 Processor -- start 2024-04-26T20:32:03.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e6c00640 1 -- start start 2024-04-26T20:32:03.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e6c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2e0108d90 0x7ff2e019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:03.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e6c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2e0109f80 0x7ff2e019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:03.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e6c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e010a940 0x7ff2e01a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e6c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e01148f0 con 0x7ff2e010a940 2024-04-26T20:32:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e6c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e0114770 con 0x7ff2e0108d90 2024-04-26T20:32:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e6c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e0114a70 con 0x7ff2e0109f80 2024-04-26T20:32:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e5800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2e0108d90 0x7ff2e019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e5800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2e0108d90 0x7ff2e019ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:36260/0 (socket says 172.21.15.102:36260) 2024-04-26T20:32:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e5800640 1 -- 172.21.15.102:0/1146105446 learned_addr learned my addr 172.21.15.102:0/1146105446 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e6200640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e010a940 0x7ff2e01a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.962+0000 7ff2e4e00640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2e0109f80 0x7ff2e019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:03.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.963+0000 7ff2e5800640 1 -- 172.21.15.102:0/1146105446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2e0109f80 msgr2=0x7ff2e019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:03.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.963+0000 7ff2e5800640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2e0109f80 0x7ff2e019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:03.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.963+0000 7ff2e5800640 1 -- 172.21.15.102:0/1146105446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e010a940 msgr2=0x7ff2e01a6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:03.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.963+0000 7ff2e5800640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e010a940 0x7ff2e01a6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:03.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.963+0000 7ff2e5800640 1 -- 172.21.15.102:0/1146105446 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e01a6ba0 con 0x7ff2e0108d90 2024-04-26T20:32:03.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.963+0000 7ff2e5800640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2e0108d90 0x7ff2e019ef80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400d950 tx=0x7ff2d400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:03.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.963+0000 7ff2de000640 1 -- 172.21.15.102:0/1146105446 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4017020 con 0x7ff2e0108d90 2024-04-26T20:32:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.963+0000 7ff2de000640 1 -- 172.21.15.102:0/1146105446 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2d4009c30 con 0x7ff2e0108d90 2024-04-26T20:32:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.963+0000 7ff2de000640 1 -- 172.21.15.102:0/1146105446 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d400f690 con 0x7ff2e0108d90 2024-04-26T20:32:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.963+0000 7ff2e6c00640 1 -- 172.21.15.102:0/1146105446 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2e01a6e90 con 0x7ff2e0108d90 2024-04-26T20:32:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.964+0000 7ff2e6c00640 1 -- 172.21.15.102:0/1146105446 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2e01a7300 con 0x7ff2e0108d90 2024-04-26T20:32:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.965+0000 7ff2e6c00640 1 -- 172.21.15.102:0/1146105446 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2e0110180 con 0x7ff2e0108d90 2024-04-26T20:32:03.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.967+0000 7ff2de000640 1 -- 172.21.15.102:0/1146105446 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ff2d401a020 con 0x7ff2e0108d90 2024-04-26T20:32:03.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.968+0000 7ff2de000640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff2c80a3c30 0x7ff2c80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:03.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.968+0000 7ff2de000640 1 -- 172.21.15.102:0/1146105446 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff2d40cfe20 con 0x7ff2e0108d90 2024-04-26T20:32:03.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.968+0000 7ff2e4e00640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff2c80a3c30 0x7ff2c80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:03.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.969+0000 7ff2e4e00640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff2c80a3c30 0x7ff2c80a60f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc031500 tx=0x7ff2cc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:03.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:03.972+0000 7ff2de000640 1 -- 172.21.15.102:0/1146105446 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff2d4021020 con 0x7ff2e0108d90 2024-04-26T20:32:04.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.213+0000 7ff2e6c00640 1 -- 172.21.15.102:0/1146105446 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2e002dad0 con 0x7ff2e0108d90 2024-04-26T20:32:04.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.214+0000 7ff2de000640 1 -- 172.21.15.102:0/1146105446 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6361 (secure 0 0 0) 0x7ff2e002dad0 con 0x7ff2e0108d90 2024-04-26T20:32:04.216 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:04.216 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T20:32:04.216 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2024-04-26T20:32:04.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 -- 172.21.15.102:0/1146105446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff2c80a3c30 msgr2=0x7ff2c80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff2c80a3c30 0x7ff2c80a60f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc031500 tx=0x7ff2cc005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 -- 172.21.15.102:0/1146105446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2e0108d90 msgr2=0x7ff2e019ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2e0108d90 0x7ff2e019ef80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400d950 tx=0x7ff2d400de10 comp rx=0 tx=0).stop 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 -- 172.21.15.102:0/1146105446 shutdown_connections 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff2c80a3c30 0x7ff2c80a60f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2e010a940 0x7ff2e01a6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2e0109f80 0x7ff2e019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 --2- 172.21.15.102:0/1146105446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff2e0108d90 0x7ff2e019ef80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 -- 172.21.15.102:0/1146105446 >> 172.21.15.102:0/1146105446 conn(0x7ff2e0077d30 msgr2=0x7ff2e010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 -- 172.21.15.102:0/1146105446 shutdown_connections 2024-04-26T20:32:04.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:04.217+0000 7ff2e6c00640 1 -- 172.21.15.102:0/1146105446 wait complete. 2024-04-26T20:32:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:04 smithi112 ceph-mon[32769]: Deploying daemon mds.h on smithi112 2024-04-26T20:32:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:04 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] up:boot 2024-04-26T20:32:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:04 smithi112 ceph-mon[32769]: fsmap 7 up:standby 2024-04-26T20:32:04.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:04 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:32:04.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:04 smithi130 ceph-mon[32704]: Deploying daemon mds.h on smithi112 2024-04-26T20:32:04.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:04 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] up:boot 2024-04-26T20:32:04.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:04 smithi130 ceph-mon[32704]: fsmap 7 up:standby 2024-04-26T20:32:04.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:04 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:32:04.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:04 smithi102 ceph-mon[28094]: Deploying daemon mds.h on smithi112 2024-04-26T20:32:04.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:04 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] up:boot 2024-04-26T20:32:04.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:04 smithi102 ceph-mon[28094]: fsmap 7 up:standby 2024-04-26T20:32:04.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:04 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:32:05.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:05 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1146105446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:05.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:05 smithi112 ceph-mon[32769]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:05.527 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:32:05 smithi112 systemd[1]: Starting Ceph mds.h for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:32:05.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:05 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1146105446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:05.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:05 smithi130 ceph-mon[32704]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:05.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:05 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1146105446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:05.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:05 smithi102 ceph-mon[28094]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:05.824 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:32:05 smithi112 podman[62519]: 2024-04-26 20:32:05.474731685 +0000 UTC m=+0.018033687 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:32:05.824 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:32:05 smithi112 podman[62519]: 2024-04-26 20:32:05.604484858 +0000 UTC m=+0.147786873 container create 0d63dc159fe08018cffad509ae11467c7f684c974b407a8a0a6d4f4949cdc1d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T20:32:05.824 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:32:05 smithi112 podman[62519]: 2024-04-26 20:32:05.823965108 +0000 UTC m=+0.367267093 container init 0d63dc159fe08018cffad509ae11467c7f684c974b407a8a0a6d4f4949cdc1d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T20:32:06.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:06.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:32:05 smithi112 podman[62519]: 2024-04-26 20:32:05.826526652 +0000 UTC m=+0.369828625 container start 0d63dc159fe08018cffad509ae11467c7f684c974b407a8a0a6d4f4949cdc1d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-26T20:32:06.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:32:05 smithi112 ceph-mds[62538]: -- 172.21.15.112:0/4291726732 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627b9135a40 con 0x5627b9109000 2024-04-26T20:32:06.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:32:05 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h[62534]: starting mds.h at 2024-04-26T20:32:06.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:32:05 smithi112 bash[62519]: 0d63dc159fe08018cffad509ae11467c7f684c974b407a8a0a6d4f4949cdc1d8 2024-04-26T20:32:06.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:32:05 smithi112 systemd[1]: Started Ceph mds.h for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:32:06.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.449+0000 7fe11b200640 1 -- 172.21.15.102:0/3958124939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe114078530 msgr2=0x7fe114078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:06.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.449+0000 7fe11b200640 1 --2- 172.21.15.102:0/3958124939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe114078530 0x7fe114078990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe104009900 tx=0x7fe10402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:06.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.449+0000 7fe11b200640 1 -- 172.21.15.102:0/3958124939 shutdown_connections 2024-04-26T20:32:06.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.449+0000 7fe11b200640 1 --2- 172.21.15.102:0/3958124939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe114078ed0 0x7fe114079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:06.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.449+0000 7fe11b200640 1 --2- 172.21.15.102:0/3958124939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe114078530 0x7fe114078990 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:06.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.449+0000 7fe11b200640 1 --2- 172.21.15.102:0/3958124939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1140772f0 0x7fe114077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:06.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.449+0000 7fe11b200640 1 -- 172.21.15.102:0/3958124939 >> 172.21.15.102:0/3958124939 conn(0x7fe1140ffeb0 msgr2=0x7fe114102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:06.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.449+0000 7fe11b200640 1 -- 172.21.15.102:0/3958124939 shutdown_connections 2024-04-26T20:32:06.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.449+0000 7fe11b200640 1 -- 172.21.15.102:0/3958124939 wait complete. 2024-04-26T20:32:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.450+0000 7fe11b200640 1 Processor -- start 2024-04-26T20:32:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.450+0000 7fe11b200640 1 -- start start 2024-04-26T20:32:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.450+0000 7fe11b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1140772f0 0x7fe11419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.450+0000 7fe11b200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe114078530 0x7fe11419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe11b200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe114078ed0 0x7fe1141a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe11b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe1141148a0 con 0x7fe1140772f0 2024-04-26T20:32:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe11b200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe114114720 con 0x7fe114078530 2024-04-26T20:32:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe11b200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe114114a20 con 0x7fe114078ed0 2024-04-26T20:32:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe114078530 0x7fe11419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe114078530 0x7fe11419f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:45800/0 (socket says 172.21.15.102:45800) 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119400640 1 -- 172.21.15.102:0/3766745114 learned_addr learned my addr 172.21.15.102:0/3766745114 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe11a800640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe114078ed0 0x7fe1141a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119e00640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1140772f0 0x7fe11419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119400640 1 -- 172.21.15.102:0/3766745114 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe114078ed0 msgr2=0x7fe1141a6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119400640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe114078ed0 0x7fe1141a6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119400640 1 -- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1140772f0 msgr2=0x7fe11419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119400640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1140772f0 0x7fe11419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119400640 1 -- 172.21.15.102:0/3766745114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1141a6b90 con 0x7fe114078530 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119e00640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1140772f0 0x7fe11419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.451+0000 7fe119400640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe114078530 0x7fe11419f4b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe104005ab0 tx=0x7fe104003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.452+0000 7fe10ea00640 1 -- 172.21.15.102:0/3766745114 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10403f070 con 0x7fe114078530 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.452+0000 7fe10ea00640 1 -- 172.21.15.102:0/3766745114 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe104031c50 con 0x7fe114078530 2024-04-26T20:32:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.452+0000 7fe10ea00640 1 -- 172.21.15.102:0/3766745114 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104035030 con 0x7fe114078530 2024-04-26T20:32:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.452+0000 7fe11b200640 1 -- 172.21.15.102:0/3766745114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1141a6e20 con 0x7fe114078530 2024-04-26T20:32:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.452+0000 7fe11b200640 1 -- 172.21.15.102:0/3766745114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1141a7230 con 0x7fe114078530 2024-04-26T20:32:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.453+0000 7fe11b200640 1 -- 172.21.15.102:0/3766745114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1141112d0 con 0x7fe114078530 2024-04-26T20:32:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.455+0000 7fe10ea00640 1 -- 172.21.15.102:0/3766745114 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe104042070 con 0x7fe114078530 2024-04-26T20:32:06.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.456+0000 7fe10ea00640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe0fc0a3c30 0x7fe0fc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:06.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.456+0000 7fe10ea00640 1 -- 172.21.15.102:0/3766745114 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe1040f06f0 con 0x7fe114078530 2024-04-26T20:32:06.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.457+0000 7fe119e00640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe0fc0a3c30 0x7fe0fc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:06.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.457+0000 7fe119e00640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe0fc0a3c30 0x7fe0fc0a60f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe108009c20 tx=0x7fe108005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:06.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.460+0000 7fe10ea00640 1 -- 172.21.15.102:0/3766745114 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe1040bd4d0 con 0x7fe114078530 2024-04-26T20:32:06.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.705+0000 7fe11b200640 1 -- 172.21.15.102:0/3766745114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe11402dcf0 con 0x7fe114078530 2024-04-26T20:32:06.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.707+0000 7fe10ea00640 1 -- 172.21.15.102:0/3766745114 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7fe1040bcdd0 con 0x7fe114078530 2024-04-26T20:32:06.708 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:06.708 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T20:32:06.708 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 9 2024-04-26T20:32:06.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.709+0000 7fe11b200640 1 -- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe0fc0a3c30 msgr2=0x7fe0fc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.709+0000 7fe11b200640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe0fc0a3c30 0x7fe0fc0a60f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe108009c20 tx=0x7fe108005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.709+0000 7fe11b200640 1 -- 172.21.15.102:0/3766745114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe114078530 msgr2=0x7fe11419f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.709+0000 7fe11b200640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe114078530 0x7fe11419f4b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe104005ab0 tx=0x7fe104003b90 comp rx=0 tx=0).stop 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.710+0000 7fe11b200640 1 -- 172.21.15.102:0/3766745114 shutdown_connections 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.710+0000 7fe11b200640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe0fc0a3c30 0x7fe0fc0a60f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.710+0000 7fe11b200640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe114078ed0 0x7fe1141a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.710+0000 7fe11b200640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe114078530 0x7fe11419f4b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.710+0000 7fe11b200640 1 --2- 172.21.15.102:0/3766745114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1140772f0 0x7fe11419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.710+0000 7fe11b200640 1 -- 172.21.15.102:0/3766745114 >> 172.21.15.102:0/3766745114 conn(0x7fe1140ffeb0 msgr2=0x7fe1141019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.710+0000 7fe11b200640 1 -- 172.21.15.102:0/3766745114 shutdown_connections 2024-04-26T20:32:06.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:06.710+0000 7fe11b200640 1 -- 172.21.15.102:0/3766745114 wait complete. 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: Deploying daemon mds.i on smithi130 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] up:boot 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: fsmap 8 up:standby 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:07.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:06 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3766745114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:07.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:07.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:07.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:07.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:07.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:07.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:07.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: Deploying daemon mds.i on smithi130 2024-04-26T20:32:07.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] up:boot 2024-04-26T20:32:07.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: fsmap 8 up:standby 2024-04-26T20:32:07.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:32:07.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:07.338 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:06 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3766745114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: Deploying daemon mds.i on smithi130 2024-04-26T20:32:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] up:boot 2024-04-26T20:32:07.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: fsmap 8 up:standby 2024-04-26T20:32:07.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:32:07.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:07.373 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:06 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3766745114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:08.372 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:32:08 smithi130 systemd[1]: Starting Ceph mds.i for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:32:08.680 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:32:08 smithi130 podman[63647]: 2024-04-26 20:32:08.321292554 +0000 UTC m=+0.017917338 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:32:08.680 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:32:08 smithi130 podman[63647]: 2024-04-26 20:32:08.433901204 +0000 UTC m=+0.130525979 container create 140f6230e432d7b006a3c585cbc450248d40cfbe47d0284ca63d8de99e93fafe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-26T20:32:08.680 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:32:08 smithi130 podman[63647]: 2024-04-26 20:32:08.680434442 +0000 UTC m=+0.377059217 container init 140f6230e432d7b006a3c585cbc450248d40cfbe47d0284ca63d8de99e93fafe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T20:32:08.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:08.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.946+0000 7fe659c00640 1 -- 172.21.15.102:0/2481374199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe654105d10 msgr2=0x7fe654108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.946+0000 7fe659c00640 1 --2- 172.21.15.102:0/2481374199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe654105d10 0x7fe654108100 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe640009900 tx=0x7fe64002a410 comp rx=0 tx=0).stop 2024-04-26T20:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.946+0000 7fe659c00640 1 -- 172.21.15.102:0/2481374199 shutdown_connections 2024-04-26T20:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.946+0000 7fe659c00640 1 --2- 172.21.15.102:0/2481374199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe654105d10 0x7fe654108100 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.946+0000 7fe659c00640 1 --2- 172.21.15.102:0/2481374199 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe654108640 0x7fe65410aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.946+0000 7fe659c00640 1 --2- 172.21.15.102:0/2481374199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6541033e0 0x7fe6541057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.946+0000 7fe659c00640 1 -- 172.21.15.102:0/2481374199 >> 172.21.15.102:0/2481374199 conn(0x7fe6540fcdf0 msgr2=0x7fe6540ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.946+0000 7fe659c00640 1 -- 172.21.15.102:0/2481374199 shutdown_connections 2024-04-26T20:32:08.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.947+0000 7fe659c00640 1 -- 172.21.15.102:0/2481374199 wait complete. 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.947+0000 7fe659c00640 1 Processor -- start 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe659c00640 1 -- start start 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe659c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6541033e0 0x7fe654102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe659c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe654105d10 0x7fe654101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe659c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe654108640 0x7fe6541018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe659c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410e4f0 con 0x7fe6541033e0 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe659c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410e370 con 0x7fe654105d10 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe659c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410e670 con 0x7fe654108640 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe652a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6541033e0 0x7fe654102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe652000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe654105d10 0x7fe654101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe652000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe654105d10 0x7fe654101390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:45830/0 (socket says 172.21.15.102:45830) 2024-04-26T20:32:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe652000640 1 -- 172.21.15.102:0/1836144762 learned_addr learned my addr 172.21.15.102:0/1836144762 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.948+0000 7fe653400640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe654108640 0x7fe6541018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.949+0000 7fe652000640 1 -- 172.21.15.102:0/1836144762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe654108640 msgr2=0x7fe6541018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.949+0000 7fe652000640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe654108640 0x7fe6541018d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.949+0000 7fe652000640 1 -- 172.21.15.102:0/1836144762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6541033e0 msgr2=0x7fe654102d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.949+0000 7fe652000640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6541033e0 0x7fe654102d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.949+0000 7fe652000640 1 -- 172.21.15.102:0/1836144762 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe654102160 con 0x7fe654105d10 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.949+0000 7fe652000640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe654105d10 0x7fe654101390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe640031500 tx=0x7fe640003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.949+0000 7fe64b400640 1 -- 172.21.15.102:0/1836144762 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64003f070 con 0x7fe654105d10 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.949+0000 7fe64b400640 1 -- 172.21.15.102:0/1836144762 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe640031c50 con 0x7fe654105d10 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.949+0000 7fe659c00640 1 -- 172.21.15.102:0/1836144762 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6541a2c80 con 0x7fe654105d10 2024-04-26T20:32:08.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.949+0000 7fe64b400640 1 -- 172.21.15.102:0/1836144762 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640035030 con 0x7fe654105d10 2024-04-26T20:32:08.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.950+0000 7fe659c00640 1 -- 172.21.15.102:0/1836144762 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6541a31d0 con 0x7fe654105d10 2024-04-26T20:32:08.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.951+0000 7fe659c00640 1 -- 172.21.15.102:0/1836144762 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6540fe8e0 con 0x7fe654105d10 2024-04-26T20:32:08.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.952+0000 7fe64b400640 1 -- 172.21.15.102:0/1836144762 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe640042070 con 0x7fe654105d10 2024-04-26T20:32:08.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.954+0000 7fe64b400640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6380a3bb0 0x7fe6380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:08.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.954+0000 7fe64b400640 1 -- 172.21.15.102:0/1836144762 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe6400f06c0 con 0x7fe654105d10 2024-04-26T20:32:08.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.954+0000 7fe652a00640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6380a3bb0 0x7fe6380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:08.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.955+0000 7fe652a00640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6380a3bb0 0x7fe6380a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe644009bd0 tx=0x7fe644005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:08.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:08.958+0000 7fe64b400640 1 -- 172.21.15.102:0/1836144762 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe6400bce50 con 0x7fe654105d10 2024-04-26T20:32:09.122 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:32:08 smithi130 podman[63647]: 2024-04-26 20:32:08.683419497 +0000 UTC m=+0.380044273 container start 140f6230e432d7b006a3c585cbc450248d40cfbe47d0284ca63d8de99e93fafe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T20:32:09.122 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:32:08 smithi130 ceph-mds[63666]: -- 172.21.15.130:0/3354963048 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560e43e33a40 con 0x560e43e06c00 2024-04-26T20:32:09.122 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:32:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i[63662]: starting mds.i at 2024-04-26T20:32:09.122 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:32:08 smithi130 bash[63647]: 140f6230e432d7b006a3c585cbc450248d40cfbe47d0284ca63d8de99e93fafe 2024-04-26T20:32:09.122 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:32:08 smithi130 systemd[1]: Started Ceph mds.i for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:32:09.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.212+0000 7fe659c00640 1 -- 172.21.15.102:0/1836144762 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe654063530 con 0x7fe654105d10 2024-04-26T20:32:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.214+0000 7fe64b400640 1 -- 172.21.15.102:0/1836144762 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7fe6400c1150 con 0x7fe654105d10 2024-04-26T20:32:09.215 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:09.215 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T20:32:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 9 2024-04-26T20:32:09.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.216+0000 7fe659c00640 1 -- 172.21.15.102:0/1836144762 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6380a3bb0 msgr2=0x7fe6380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:09.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.216+0000 7fe659c00640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6380a3bb0 0x7fe6380a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe644009bd0 tx=0x7fe644005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:09.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.216+0000 7fe659c00640 1 -- 172.21.15.102:0/1836144762 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe654105d10 msgr2=0x7fe654101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:09.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.216+0000 7fe659c00640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe654105d10 0x7fe654101390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe640031500 tx=0x7fe640003c10 comp rx=0 tx=0).stop 2024-04-26T20:32:09.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.216+0000 7fe659c00640 1 -- 172.21.15.102:0/1836144762 shutdown_connections 2024-04-26T20:32:09.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.216+0000 7fe659c00640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6380a3bb0 0x7fe6380a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:09.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.216+0000 7fe659c00640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe654108640 0x7fe6541018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:09.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.216+0000 7fe659c00640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe654105d10 0x7fe654101390 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:09.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.216+0000 7fe659c00640 1 --2- 172.21.15.102:0/1836144762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6541033e0 0x7fe654102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:09.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.216+0000 7fe659c00640 1 -- 172.21.15.102:0/1836144762 >> 172.21.15.102:0/1836144762 conn(0x7fe6540fcdf0 msgr2=0x7fe654103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:09.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.217+0000 7fe659c00640 1 -- 172.21.15.102:0/1836144762 shutdown_connections 2024-04-26T20:32:09.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:09.217+0000 7fe659c00640 1 -- 172.21.15.102:0/1836144762 wait complete. 2024-04-26T20:32:09.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:09 smithi102 ceph-mon[28094]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:09.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:09.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:09.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:09.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:09.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:09.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:09 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:09.841 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:09 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1836144762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:09 smithi130 ceph-mon[32704]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:09 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:09 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:09 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:09 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:09 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:09 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:09.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:09 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1836144762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:10.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:09 smithi112 ceph-mon[32769]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:09 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:09 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:09 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:09 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:09 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:09 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:09 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1836144762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:10.849 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:10 smithi130 ceph-mon[32704]: Deploying daemon mds.g on smithi102 2024-04-26T20:32:10.849 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:10 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] up:boot 2024-04-26T20:32:10.849 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:10 smithi130 ceph-mon[32704]: fsmap 9 up:standby 2024-04-26T20:32:10.849 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:10 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:32:10.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:10 smithi102 ceph-mon[28094]: Deploying daemon mds.g on smithi102 2024-04-26T20:32:10.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:10 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] up:boot 2024-04-26T20:32:10.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:10 smithi102 ceph-mon[28094]: fsmap 9 up:standby 2024-04-26T20:32:10.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:10 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:32:11.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:10 smithi112 ceph-mon[32769]: Deploying daemon mds.g on smithi102 2024-04-26T20:32:11.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:10 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] up:boot 2024-04-26T20:32:11.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:10 smithi112 ceph-mon[32769]: fsmap 9 up:standby 2024-04-26T20:32:11.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:10 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:32:11.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.457+0000 7f816e800640 1 -- 172.21.15.102:0/2091939219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8168102970 msgr2=0x7f8168102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.457+0000 7f816e800640 1 --2- 172.21.15.102:0/2091939219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8168102970 0x7f8168102d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8158009900 tx=0x7f815802a410 comp rx=0 tx=0).stop 2024-04-26T20:32:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.457+0000 7f816e800640 1 -- 172.21.15.102:0/2091939219 shutdown_connections 2024-04-26T20:32:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.457+0000 7f816e800640 1 --2- 172.21.15.102:0/2091939219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8168104520 0x7f816810adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.457+0000 7f816e800640 1 --2- 172.21.15.102:0/2091939219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8168103b60 0x7f8168103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.457+0000 7f816e800640 1 --2- 172.21.15.102:0/2091939219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8168102970 0x7f8168102d90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.457+0000 7f816e800640 1 -- 172.21.15.102:0/2091939219 >> 172.21.15.102:0/2091939219 conn(0x7f81680fdee0 msgr2=0x7f8168100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.457+0000 7f816e800640 1 -- 172.21.15.102:0/2091939219 shutdown_connections 2024-04-26T20:32:11.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.457+0000 7f816e800640 1 -- 172.21.15.102:0/2091939219 wait complete. 2024-04-26T20:32:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.458+0000 7f816e800640 1 Processor -- start 2024-04-26T20:32:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f816e800640 1 -- start start 2024-04-26T20:32:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f816e800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8168102970 0x7f8168198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f816e800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8168103b60 0x7f81681990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f816e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8168104520 0x7f81681a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f816e800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f816810e4e0 con 0x7f8168104520 2024-04-26T20:32:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f816e800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f816810e360 con 0x7f8168103b60 2024-04-26T20:32:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f816e800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f816810e660 con 0x7f8168102970 2024-04-26T20:32:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f815ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8168103b60 0x7f81681990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:11.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f8167400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8168102970 0x7f8168198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f815ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8168103b60 0x7f81681990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:45852/0 (socket says 172.21.15.102:45852) 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f815ea00640 1 -- 172.21.15.102:0/2257626457 learned_addr learned my addr 172.21.15.102:0/2257626457 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f8167400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8168102970 0x7f8168198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:53136/0 (socket says 172.21.15.102:53136) 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.459+0000 7f8167e00640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8168104520 0x7f81681a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.460+0000 7f815ea00640 1 -- 172.21.15.102:0/2257626457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8168102970 msgr2=0x7f8168198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.460+0000 7f815ea00640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8168102970 0x7f8168198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.460+0000 7f815ea00640 1 -- 172.21.15.102:0/2257626457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8168104520 msgr2=0x7f81681a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.460+0000 7f815ea00640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8168104520 0x7f81681a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.460+0000 7f815ea00640 1 -- 172.21.15.102:0/2257626457 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81681a07d0 con 0x7f8168103b60 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.460+0000 7f815ea00640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8168103b60 0x7f81681990f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f815000e9f0 tx=0x7f815000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.460+0000 7f8164c00640 1 -- 172.21.15.102:0/2257626457 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815000cd80 con 0x7f8168103b60 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.460+0000 7f816e800640 1 -- 172.21.15.102:0/2257626457 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81681a0ac0 con 0x7f8168103b60 2024-04-26T20:32:11.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.460+0000 7f8164c00640 1 -- 172.21.15.102:0/2257626457 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81500054d0 con 0x7f8168103b60 2024-04-26T20:32:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.460+0000 7f8164c00640 1 -- 172.21.15.102:0/2257626457 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8150010690 con 0x7f8168103b60 2024-04-26T20:32:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.461+0000 7f816e800640 1 -- 172.21.15.102:0/2257626457 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81681a1010 con 0x7f8168103b60 2024-04-26T20:32:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.462+0000 7f816e800640 1 -- 172.21.15.102:0/2257626457 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8168109d10 con 0x7f8168103b60 2024-04-26T20:32:11.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.464+0000 7f8164c00640 1 -- 172.21.15.102:0/2257626457 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f8150004e90 con 0x7f8168103b60 2024-04-26T20:32:11.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.466+0000 7f8164c00640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f814c0a3c30 0x7f814c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.466+0000 7f8164c00640 1 -- 172.21.15.102:0/2257626457 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f81500d0340 con 0x7f8168103b60 2024-04-26T20:32:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.466+0000 7f8167400640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f814c0a3c30 0x7f814c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:11.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.467+0000 7f8167400640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f814c0a3c30 0x7f814c0a60f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8158005ab0 tx=0x7f8158005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:11.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.467+0000 7f8164c00640 1 -- 172.21.15.102:0/2257626457 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f815009d120 con 0x7f8168103b60 2024-04-26T20:32:11.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.726+0000 7f816e800640 1 -- 172.21.15.102:0/2257626457 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f816802dad0 con 0x7f8168103b60 2024-04-26T20:32:11.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.727+0000 7f8164c00640 1 -- 172.21.15.102:0/2257626457 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7f815009ca20 con 0x7f8168103b60 2024-04-26T20:32:11.728 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:11.728 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T20:32:11.729 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2024-04-26T20:32:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.729+0000 7f816e800640 1 -- 172.21.15.102:0/2257626457 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f814c0a3c30 msgr2=0x7f814c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.729+0000 7f816e800640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f814c0a3c30 0x7f814c0a60f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8158005ab0 tx=0x7f8158005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.729+0000 7f816e800640 1 -- 172.21.15.102:0/2257626457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8168103b60 msgr2=0x7f81681990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.729+0000 7f816e800640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8168103b60 0x7f81681990f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f815000e9f0 tx=0x7f815000edf0 comp rx=0 tx=0).stop 2024-04-26T20:32:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.730+0000 7f816e800640 1 -- 172.21.15.102:0/2257626457 shutdown_connections 2024-04-26T20:32:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.730+0000 7f816e800640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f814c0a3c30 0x7f814c0a60f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:11.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.730+0000 7f816e800640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8168104520 0x7f81681a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.730+0000 7f816e800640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8168103b60 0x7f81681990f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.730+0000 7f816e800640 1 --2- 172.21.15.102:0/2257626457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8168102970 0x7f8168198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.730+0000 7f816e800640 1 -- 172.21.15.102:0/2257626457 >> 172.21.15.102:0/2257626457 conn(0x7f81680fdee0 msgr2=0x7f8168106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.730+0000 7f816e800640 1 -- 172.21.15.102:0/2257626457 shutdown_connections 2024-04-26T20:32:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:11.730+0000 7f816e800640 1 -- 172.21.15.102:0/2257626457 wait complete. 2024-04-26T20:32:11.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:11 smithi130 ceph-mon[32704]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:11.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:11 smithi102 ceph-mon[28094]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:12.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:11 smithi112 ceph-mon[32769]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:12.352 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:32:11 smithi102 systemd[1]: Starting Ceph mds.g for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:32:12.606 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:32:12 smithi102 podman[69102]: 2024-04-26 20:32:12.256427215 +0000 UTC m=+0.031858589 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:32:12.606 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:32:12 smithi102 podman[69102]: 2024-04-26 20:32:12.404707606 +0000 UTC m=+0.180139028 container create facab8299a722fec42fa8b44cfbde018c89c74c8240fed0ec2295e660cb46dd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:32:12.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:12 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2257626457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:12.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:12 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2257626457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:12.899 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:32:12 smithi102 podman[69102]: 2024-04-26 20:32:12.721202373 +0000 UTC m=+0.496633772 container init facab8299a722fec42fa8b44cfbde018c89c74c8240fed0ec2295e660cb46dd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-26T20:32:12.899 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:32:12 smithi102 podman[69102]: 2024-04-26 20:32:12.73164668 +0000 UTC m=+0.507078079 container start facab8299a722fec42fa8b44cfbde018c89c74c8240fed0ec2295e660cb46dd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:32:12.899 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:32:12 smithi102 ceph-mds[69121]: -- 172.21.15.102:0/2952972560 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563f0fba5a40 con 0x563f0fb78c00 2024-04-26T20:32:12.899 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:32:12 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g[69117]: starting mds.g at 2024-04-26T20:32:12.899 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:32:12 smithi102 bash[69102]: facab8299a722fec42fa8b44cfbde018c89c74c8240fed0ec2295e660cb46dd7 2024-04-26T20:32:12.899 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:32:12 smithi102 systemd[1]: Started Ceph mds.g for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:32:13.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:12 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2257626457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:13.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:13.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:13 smithi130 ceph-mon[32704]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:13.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:13.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:13.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:13.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:13.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:13 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:13.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:13 smithi102 ceph-mon[28094]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:13.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:13.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:13.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:13.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:13.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:13.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:13 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:13.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.945+0000 7f6fafc00640 1 -- 172.21.15.102:0/2540170251 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6fa8104b90 msgr2=0x7f6fa8104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:13.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.945+0000 7f6fafc00640 1 --2- 172.21.15.102:0/2540170251 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6fa8104b90 0x7f6fa8104fb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009be0 tx=0x7f6f9c02a6c0 comp rx=0 tx=0).stop 2024-04-26T20:32:13.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.945+0000 7f6fafc00640 1 -- 172.21.15.102:0/2540170251 shutdown_connections 2024-04-26T20:32:13.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.945+0000 7f6fafc00640 1 --2- 172.21.15.102:0/2540170251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fa8106740 0x7f6fa810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:13.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.945+0000 7f6fafc00640 1 --2- 172.21.15.102:0/2540170251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fa8105d80 0x7f6fa8106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:13.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.945+0000 7f6fafc00640 1 --2- 172.21.15.102:0/2540170251 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6fa8104b90 0x7f6fa8104fb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:13.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.945+0000 7f6fafc00640 1 -- 172.21.15.102:0/2540170251 >> 172.21.15.102:0/2540170251 conn(0x7f6fa8100120 msgr2=0x7f6fa8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:13.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.945+0000 7f6fafc00640 1 -- 172.21.15.102:0/2540170251 shutdown_connections 2024-04-26T20:32:13.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.945+0000 7f6fafc00640 1 -- 172.21.15.102:0/2540170251 wait complete. 2024-04-26T20:32:13.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.946+0000 7f6fafc00640 1 Processor -- start 2024-04-26T20:32:13.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.946+0000 7f6fafc00640 1 -- start start 2024-04-26T20:32:13.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fafc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fa8104b90 0x7f6fa819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:13.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fafc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6fa8105d80 0x7f6fa819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fafc00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fa8106740 0x7f6fa81a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fafc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa81106b0 con 0x7f6fa8104b90 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fafc00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa8110530 con 0x7f6fa8105d80 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fafc00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa8110830 con 0x7f6fa8106740 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fad200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fa8104b90 0x7f6fa819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fad200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fa8104b90 0x7f6fa819ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50778/0 (socket says 172.21.15.102:50778) 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fad200640 1 -- 172.21.15.102:0/1867944756 learned_addr learned my addr 172.21.15.102:0/1867944756 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fa7e00640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6fa8105d80 0x7f6fa819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.947+0000 7f6fadc00640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fa8106740 0x7f6fa81a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fad200640 1 -- 172.21.15.102:0/1867944756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fa8106740 msgr2=0x7f6fa81a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fad200640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fa8106740 0x7f6fa81a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fad200640 1 -- 172.21.15.102:0/1867944756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6fa8105d80 msgr2=0x7f6fa819b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:13.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fad200640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6fa8105d80 0x7f6fa819b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:13.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fad200640 1 -- 172.21.15.102:0/1867944756 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa81a2a00 con 0x7f6fa8104b90 2024-04-26T20:32:13.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fad200640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fa8104b90 0x7f6fa819ade0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c000f80 tx=0x7f6f9c030d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:13.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fa5600640 1 -- 172.21.15.102:0/1867944756 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c041020 con 0x7f6fa8104b90 2024-04-26T20:32:13.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fafc00640 1 -- 172.21.15.102:0/1867944756 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa81a2c90 con 0x7f6fa8104b90 2024-04-26T20:32:13.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fa5600640 1 -- 172.21.15.102:0/1867944756 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f9c0318f0 con 0x7f6fa8104b90 2024-04-26T20:32:13.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fa5600640 1 -- 172.21.15.102:0/1867944756 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c036040 con 0x7f6fa8104b90 2024-04-26T20:32:13.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.948+0000 7f6fafc00640 1 -- 172.21.15.102:0/1867944756 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa81a31d0 con 0x7f6fa8104b90 2024-04-26T20:32:13.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.950+0000 7f6fafc00640 1 -- 172.21.15.102:0/1867944756 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa810bf80 con 0x7f6fa8104b90 2024-04-26T20:32:13.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.952+0000 7f6fa5600640 1 -- 172.21.15.102:0/1867944756 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f6f9c044020 con 0x7f6fa8104b90 2024-04-26T20:32:13.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.953+0000 7f6fa5600640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f900a3bb0 0x7f6f900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:13.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.954+0000 7f6fa5600640 1 -- 172.21.15.102:0/1867944756 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6f9c0f0ff0 con 0x7f6fa8104b90 2024-04-26T20:32:13.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.954+0000 7f6fa7e00640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f900a3bb0 0x7f6f900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:13.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.954+0000 7f6fa7e00640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f900a3bb0 0x7f6f900a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6fa819c2f0 tx=0x7f6f94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:13.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:13.956+0000 7f6fa5600640 1 -- 172.21.15.102:0/1867944756 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6f9c04b020 con 0x7f6fa8104b90 2024-04-26T20:32:14.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:13 smithi112 ceph-mon[32769]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:13 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:14.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.201+0000 7f6fafc00640 1 -- 172.21.15.102:0/1867944756 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fa802dad0 con 0x7f6fa8104b90 2024-04-26T20:32:14.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.202+0000 7f6fa5600640 1 -- 172.21.15.102:0/1867944756 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8842 (secure 0 0 0) 0x7f6f9c0bde30 con 0x7f6fa8104b90 2024-04-26T20:32:14.204 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:14.204 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T20:32:14.205 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 11 2024-04-26T20:32:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.205+0000 7f6fafc00640 1 -- 172.21.15.102:0/1867944756 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f900a3bb0 msgr2=0x7f6f900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.205+0000 7f6fafc00640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f900a3bb0 0x7f6f900a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6fa819c2f0 tx=0x7f6f94005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.205+0000 7f6fafc00640 1 -- 172.21.15.102:0/1867944756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fa8104b90 msgr2=0x7f6fa819ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:14.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.205+0000 7f6fafc00640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fa8104b90 0x7f6fa819ade0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c000f80 tx=0x7f6f9c030d10 comp rx=0 tx=0).stop 2024-04-26T20:32:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.205+0000 7f6fafc00640 1 -- 172.21.15.102:0/1867944756 shutdown_connections 2024-04-26T20:32:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.205+0000 7f6fafc00640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f900a3bb0 0x7f6f900a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.205+0000 7f6fafc00640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fa8106740 0x7f6fa81a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.205+0000 7f6fafc00640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6fa8105d80 0x7f6fa819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:14.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.205+0000 7f6fafc00640 1 --2- 172.21.15.102:0/1867944756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fa8104b90 0x7f6fa819ade0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:14.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.205+0000 7f6fafc00640 1 -- 172.21.15.102:0/1867944756 >> 172.21.15.102:0/1867944756 conn(0x7f6fa8100120 msgr2=0x7f6fa8109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:14.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.206+0000 7f6fafc00640 1 -- 172.21.15.102:0/1867944756 shutdown_connections 2024-04-26T20:32:14.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:14.206+0000 7f6fafc00640 1 -- 172.21.15.102:0/1867944756 wait complete. 2024-04-26T20:32:14.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:14 smithi130 ceph-mon[32704]: Deploying daemon mds.a on smithi102 2024-04-26T20:32:14.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:14 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] up:boot 2024-04-26T20:32:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:14 smithi130 ceph-mon[32704]: fsmap 10 up:standby 2024-04-26T20:32:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:14 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T20:32:14.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:14 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1867944756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:14 smithi102 ceph-mon[28094]: Deploying daemon mds.a on smithi102 2024-04-26T20:32:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:14 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] up:boot 2024-04-26T20:32:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:14 smithi102 ceph-mon[28094]: fsmap 10 up:standby 2024-04-26T20:32:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:14 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T20:32:14.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:14 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1867944756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:15.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:14 smithi112 ceph-mon[32769]: Deploying daemon mds.a on smithi102 2024-04-26T20:32:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:14 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] up:boot 2024-04-26T20:32:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:14 smithi112 ceph-mon[32769]: fsmap 10 up:standby 2024-04-26T20:32:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:14 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T20:32:15.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:14 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1867944756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:15.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:15 smithi102 ceph-mon[28094]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:16.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:15 smithi112 ceph-mon[32769]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:16.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:15 smithi130 ceph-mon[32704]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:16.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.428+0000 7f014fc00640 1 -- 172.21.15.102:0/3045457714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0148104b90 msgr2=0x7f0148104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.428+0000 7f014fc00640 1 --2- 172.21.15.102:0/3045457714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0148104b90 0x7f0148104fb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0138009900 tx=0x7f013802a410 comp rx=0 tx=0).stop 2024-04-26T20:32:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.428+0000 7f014fc00640 1 -- 172.21.15.102:0/3045457714 shutdown_connections 2024-04-26T20:32:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.428+0000 7f014fc00640 1 --2- 172.21.15.102:0/3045457714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0148106740 0x7f014810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.428+0000 7f014fc00640 1 --2- 172.21.15.102:0/3045457714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0148105d80 0x7f0148106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:16.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.428+0000 7f014fc00640 1 --2- 172.21.15.102:0/3045457714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0148104b90 0x7f0148104fb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.428+0000 7f014fc00640 1 -- 172.21.15.102:0/3045457714 >> 172.21.15.102:0/3045457714 conn(0x7f0148100120 msgr2=0x7f0148102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.428+0000 7f014fc00640 1 -- 172.21.15.102:0/3045457714 shutdown_connections 2024-04-26T20:32:16.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.428+0000 7f014fc00640 1 -- 172.21.15.102:0/3045457714 wait complete. 2024-04-26T20:32:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.429+0000 7f014fc00640 1 Processor -- start 2024-04-26T20:32:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014fc00640 1 -- start start 2024-04-26T20:32:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014fc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0148104b90 0x7f014819adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014fc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0148105d80 0x7f014819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014fc00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0148106740 0x7f01481a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014fc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0148110700 con 0x7f0148104b90 2024-04-26T20:32:16.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014fc00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0148110580 con 0x7f0148105d80 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014fc00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0148110880 con 0x7f0148106740 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0148104b90 0x7f014819adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0148104b90 0x7f014819adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45338/0 (socket says 172.21.15.102:45338) 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014d200640 1 -- 172.21.15.102:0/2043337980 learned_addr learned my addr 172.21.15.102:0/2043337980 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f013fe00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0148105d80 0x7f014819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.430+0000 7f014dc00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0148106740 0x7f01481a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f014d200640 1 -- 172.21.15.102:0/2043337980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0148106740 msgr2=0x7f01481a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f014d200640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0148106740 0x7f01481a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f014d200640 1 -- 172.21.15.102:0/2043337980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0148105d80 msgr2=0x7f014819b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f014d200640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0148105d80 0x7f014819b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f014d200640 1 -- 172.21.15.102:0/2043337980 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01481a29d0 con 0x7f0148104b90 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f013fe00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0148105d80 0x7f014819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:16.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f014d200640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0148104b90 0x7f014819adc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0138031500 tx=0x7f0138003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f0146000640 1 -- 172.21.15.102:0/2043337980 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013803f070 con 0x7f0148104b90 2024-04-26T20:32:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f0146000640 1 -- 172.21.15.102:0/2043337980 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0138031c50 con 0x7f0148104b90 2024-04-26T20:32:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f014fc00640 1 -- 172.21.15.102:0/2043337980 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0148076d70 con 0x7f0148104b90 2024-04-26T20:32:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f0146000640 1 -- 172.21.15.102:0/2043337980 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138035030 con 0x7f0148104b90 2024-04-26T20:32:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.431+0000 7f014fc00640 1 -- 172.21.15.102:0/2043337980 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01480772b0 con 0x7f0148104b90 2024-04-26T20:32:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.433+0000 7f014fc00640 1 -- 172.21.15.102:0/2043337980 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f014810bed0 con 0x7f0148104b90 2024-04-26T20:32:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.435+0000 7f0146000640 1 -- 172.21.15.102:0/2043337980 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f0138042070 con 0x7f0148104b90 2024-04-26T20:32:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.436+0000 7f0146000640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f012c0a3c30 0x7f012c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.436+0000 7f0146000640 1 -- 172.21.15.102:0/2043337980 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f01380f06d0 con 0x7f0148104b90 2024-04-26T20:32:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.437+0000 7f013fe00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f012c0a3c30 0x7f012c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.437+0000 7f013fe00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f012c0a3c30 0x7f012c0a60f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0130009c40 tx=0x7f0130005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.441+0000 7f0146000640 1 -- 172.21.15.102:0/2043337980 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f01380bce60 con 0x7f0148104b90 2024-04-26T20:32:16.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.703+0000 7f014fc00640 1 -- 172.21.15.102:0/2043337980 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f014802dad0 con 0x7f0148104b90 2024-04-26T20:32:16.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.704+0000 7f0146000640 1 -- 172.21.15.102:0/2043337980 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8842 (secure 0 0 0) 0x7f01380c1160 con 0x7f0148104b90 2024-04-26T20:32:16.705 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:16.706 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T20:32:16.706 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 11 2024-04-26T20:32:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.706+0000 7f014fc00640 1 -- 172.21.15.102:0/2043337980 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f012c0a3c30 msgr2=0x7f012c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.706+0000 7f014fc00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f012c0a3c30 0x7f012c0a60f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0130009c40 tx=0x7f0130005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.706+0000 7f014fc00640 1 -- 172.21.15.102:0/2043337980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0148104b90 msgr2=0x7f014819adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.706+0000 7f014fc00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0148104b90 0x7f014819adc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0138031500 tx=0x7f0138003c10 comp rx=0 tx=0).stop 2024-04-26T20:32:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.706+0000 7f014fc00640 1 -- 172.21.15.102:0/2043337980 shutdown_connections 2024-04-26T20:32:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.706+0000 7f014fc00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f012c0a3c30 0x7f012c0a60f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.706+0000 7f014fc00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0148106740 0x7f01481a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.706+0000 7f014fc00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0148105d80 0x7f014819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.706+0000 7f014fc00640 1 --2- 172.21.15.102:0/2043337980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0148104b90 0x7f014819adc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:16.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.707+0000 7f014fc00640 1 -- 172.21.15.102:0/2043337980 >> 172.21.15.102:0/2043337980 conn(0x7f0148100120 msgr2=0x7f01481090a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.707+0000 7f014fc00640 1 -- 172.21.15.102:0/2043337980 shutdown_connections 2024-04-26T20:32:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:16.707+0000 7f014fc00640 1 -- 172.21.15.102:0/2043337980 wait complete. 2024-04-26T20:32:17.399 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 26 20:32:17 smithi102 systemd[1]: Started Ceph mds.a for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:32:17.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:17 smithi112 ceph-mon[32769]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:17.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:17 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2043337980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:17.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:17.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:17.941 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:17.941 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:17.941 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:17.941 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:18.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:17 smithi130 ceph-mon[32704]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:17 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2043337980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:18.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:17 smithi102 ceph-mon[28094]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:18.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:17 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2043337980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:18.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:18.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:18.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:18.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T20:32:18.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T20:32:18.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:18.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:18.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:18 smithi112 ceph-mon[32769]: Deploying daemon mds.b on smithi112 2024-04-26T20:32:18.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:18 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.102:6840/2992258222,v1:172.21.15.102:6841/2992258222] up:boot 2024-04-26T20:32:18.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:18 smithi112 ceph-mon[32769]: fsmap 11 up:standby 2024-04-26T20:32:18.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:18 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T20:32:18.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.939+0000 7f3b9ca00640 1 -- 172.21.15.102:0/258767016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b9810ab40 msgr2=0x7f3b981113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.939+0000 7f3b9ca00640 1 --2- 172.21.15.102:0/258767016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b9810ab40 0x7f3b981113d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009ca0 tx=0x7f3b9002a8b0 comp rx=0 tx=0).stop 2024-04-26T20:32:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.939+0000 7f3b9ca00640 1 -- 172.21.15.102:0/258767016 shutdown_connections 2024-04-26T20:32:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.939+0000 7f3b9ca00640 1 --2- 172.21.15.102:0/258767016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b9810ab40 0x7f3b981113d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.939+0000 7f3b9ca00640 1 --2- 172.21.15.102:0/258767016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9810a180 0x7f3b9810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.939+0000 7f3b9ca00640 1 --2- 172.21.15.102:0/258767016 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b98108f90 0x7f3b981093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.939+0000 7f3b9ca00640 1 -- 172.21.15.102:0/258767016 >> 172.21.15.102:0/258767016 conn(0x7f3b98077f40 msgr2=0x7f3b9807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.939+0000 7f3b9ca00640 1 -- 172.21.15.102:0/258767016 shutdown_connections 2024-04-26T20:32:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.939+0000 7f3b9ca00640 1 -- 172.21.15.102:0/258767016 wait complete. 2024-04-26T20:32:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.940+0000 7f3b9ca00640 1 Processor -- start 2024-04-26T20:32:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.940+0000 7f3b9ca00640 1 -- start start 2024-04-26T20:32:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.940+0000 7f3b9ca00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b98108f90 0x7f3b9819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.940+0000 7f3b9ca00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9810a180 0x7f3b9819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.940+0000 7f3b9ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b9810ab40 0x7f3b981a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.940+0000 7f3b9ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3b98114ae0 con 0x7f3b9810ab40 2024-04-26T20:32:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.940+0000 7f3b9ca00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3b98114960 con 0x7f3b98108f90 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.940+0000 7f3b9ca00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3b98114c60 con 0x7f3b9810a180 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b95e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b98108f90 0x7f3b9819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b96800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b9810ab40 0x7f3b981a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b95e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b98108f90 0x7f3b9819f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:36448/0 (socket says 172.21.15.102:36448) 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b95e00640 1 -- 172.21.15.102:0/4218721722 learned_addr learned my addr 172.21.15.102:0/4218721722 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b8fe00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9810a180 0x7f3b9819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b95e00640 1 -- 172.21.15.102:0/4218721722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9810a180 msgr2=0x7f3b9819f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b95e00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9810a180 0x7f3b9819f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b95e00640 1 -- 172.21.15.102:0/4218721722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b9810ab40 msgr2=0x7f3b981a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b95e00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b9810ab40 0x7f3b981a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b95e00640 1 -- 172.21.15.102:0/4218721722 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b981a6db0 con 0x7f3b98108f90 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.941+0000 7f3b95e00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b98108f90 0x7f3b9819f190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800e9c0 tx=0x7f3b8800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.942+0000 7f3b8ea00640 1 -- 172.21.15.102:0/4218721722 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8800cd80 con 0x7f3b98108f90 2024-04-26T20:32:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.942+0000 7f3b8ea00640 1 -- 172.21.15.102:0/4218721722 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b880035f0 con 0x7f3b98108f90 2024-04-26T20:32:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.942+0000 7f3b9ca00640 1 -- 172.21.15.102:0/4218721722 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b981a70a0 con 0x7f3b98108f90 2024-04-26T20:32:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.942+0000 7f3b8ea00640 1 -- 172.21.15.102:0/4218721722 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b88010690 con 0x7f3b98108f90 2024-04-26T20:32:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.942+0000 7f3b9ca00640 1 -- 172.21.15.102:0/4218721722 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b981a75f0 con 0x7f3b98108f90 2024-04-26T20:32:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.943+0000 7f3b9ca00640 1 -- 172.21.15.102:0/4218721722 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b98110250 con 0x7f3b98108f90 2024-04-26T20:32:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.945+0000 7f3b8ea00640 1 -- 172.21.15.102:0/4218721722 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f3b880037a0 con 0x7f3b98108f90 2024-04-26T20:32:18.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.946+0000 7f3b8ea00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3b7c0a3bb0 0x7f3b7c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:18.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.946+0000 7f3b8ea00640 1 -- 172.21.15.102:0/4218721722 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3b880d03b0 con 0x7f3b98108f90 2024-04-26T20:32:18.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.946+0000 7f3b8fe00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3b7c0a3bb0 0x7f3b7c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:18.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.947+0000 7f3b8fe00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3b7c0a3bb0 0x7f3b7c0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3b981a06a0 tx=0x7f3b8000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:18.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:18.953+0000 7f3b8ea00640 1 -- 172.21.15.102:0/4218721722 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3b8809d190 con 0x7f3b98108f90 2024-04-26T20:32:19.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:18 smithi130 ceph-mon[32704]: Deploying daemon mds.b on smithi112 2024-04-26T20:32:19.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:18 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.102:6840/2992258222,v1:172.21.15.102:6841/2992258222] up:boot 2024-04-26T20:32:19.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:18 smithi130 ceph-mon[32704]: fsmap 11 up:standby 2024-04-26T20:32:19.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:18 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T20:32:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:18 smithi102 ceph-mon[28094]: Deploying daemon mds.b on smithi112 2024-04-26T20:32:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:18 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.102:6840/2992258222,v1:172.21.15.102:6841/2992258222] up:boot 2024-04-26T20:32:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:18 smithi102 ceph-mon[28094]: fsmap 11 up:standby 2024-04-26T20:32:19.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:18 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T20:32:19.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.188+0000 7f3b9ca00640 1 -- 172.21.15.102:0/4218721722 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b9802dad0 con 0x7f3b98108f90 2024-04-26T20:32:19.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.190+0000 7f3b8ea00640 1 -- 172.21.15.102:0/4218721722 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9671 (secure 0 0 0) 0x7f3b8809ca90 con 0x7f3b98108f90 2024-04-26T20:32:19.191 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:19.191 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T20:32:19.191 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 12 2024-04-26T20:32:19.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 -- 172.21.15.102:0/4218721722 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3b7c0a3bb0 msgr2=0x7f3b7c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:19.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3b7c0a3bb0 0x7f3b7c0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3b981a06a0 tx=0x7f3b8000b390 comp rx=0 tx=0).stop 2024-04-26T20:32:19.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 -- 172.21.15.102:0/4218721722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b98108f90 msgr2=0x7f3b9819f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:19.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b98108f90 0x7f3b9819f190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800e9c0 tx=0x7f3b8800ee80 comp rx=0 tx=0).stop 2024-04-26T20:32:19.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 -- 172.21.15.102:0/4218721722 shutdown_connections 2024-04-26T20:32:19.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f3b7c0a3bb0 0x7f3b7c0a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:19.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b9810ab40 0x7f3b981a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:19.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9810a180 0x7f3b9819f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:19.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 --2- 172.21.15.102:0/4218721722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b98108f90 0x7f3b9819f190 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:19.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 -- 172.21.15.102:0/4218721722 >> 172.21.15.102:0/4218721722 conn(0x7f3b98077f40 msgr2=0x7f3b9810d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:19.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 -- 172.21.15.102:0/4218721722 shutdown_connections 2024-04-26T20:32:19.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:19.192+0000 7f3b9ca00640 1 -- 172.21.15.102:0/4218721722 wait complete. 2024-04-26T20:32:19.527 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:19 smithi112 systemd[1]: Starting Ceph mds.b for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:32:19.922 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:19 smithi112 ceph-mon[32769]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:19.922 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:19 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/4218721722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:19.922 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:19 smithi112 podman[62841]: 2024-04-26 20:32:19.569138832 +0000 UTC m=+0.017817367 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:32:19.922 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:19 smithi112 podman[62841]: 2024-04-26 20:32:19.694176996 +0000 UTC m=+0.142855553 container create bce7f06a9fee546b2f861d5137a54cdd4b7a1c4ccba8cd4dd563bca9dcc3d8d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:32:19.922 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:19 smithi112 podman[62841]: 2024-04-26 20:32:19.922006257 +0000 UTC m=+0.370684771 container init bce7f06a9fee546b2f861d5137a54cdd4b7a1c4ccba8cd4dd563bca9dcc3d8d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T20:32:20.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:19 smithi130 ceph-mon[32704]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:20.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:19 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/4218721722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:20.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:19 smithi102 ceph-mon[28094]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:20.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:19 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/4218721722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:20.277 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:19 smithi112 podman[62841]: 2024-04-26 20:32:19.924713851 +0000 UTC m=+0.373392364 container start bce7f06a9fee546b2f861d5137a54cdd4b7a1c4ccba8cd4dd563bca9dcc3d8d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-26T20:32:20.277 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:19 smithi112 ceph-mds[62861]: -- 172.21.15.112:0/833559561 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55accf58da40 con 0x55accf560c00 2024-04-26T20:32:20.277 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:19 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b[62857]: starting mds.b at 2024-04-26T20:32:20.277 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:20 smithi112 bash[62841]: bce7f06a9fee546b2f861d5137a54cdd4b7a1c4ccba8cd4dd563bca9dcc3d8d6 2024-04-26T20:32:20.277 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:20 smithi112 systemd[1]: Started Ceph mds.b for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:32:21.212 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:21.319 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.319 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.319 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.319 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.319 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.319 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:32:21.319 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:21 smithi112 ceph-mon[32769]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:21.319 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:21 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] up:boot 2024-04-26T20:32:21.319 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:21 smithi112 ceph-mon[32769]: fsmap 12 up:standby 2024-04-26T20:32:21.320 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:32:21.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:32:21.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:21 smithi130 ceph-mon[32704]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:21.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:21 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] up:boot 2024-04-26T20:32:21.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:21 smithi130 ceph-mon[32704]: fsmap 12 up:standby 2024-04-26T20:32:21.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:32:21.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:32:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:21 smithi102 ceph-mon[28094]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:21 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] up:boot 2024-04-26T20:32:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:21 smithi102 ceph-mon[28094]: fsmap 12 up:standby 2024-04-26T20:32:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:32:21.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.410+0000 7f7a77800640 1 -- 172.21.15.102:0/3226604982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a701033e0 msgr2=0x7f7a701057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:21.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.410+0000 7f7a77800640 1 --2- 172.21.15.102:0/3226604982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a701033e0 0x7f7a701057d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009900 tx=0x7f7a6402a410 comp rx=0 tx=0).stop 2024-04-26T20:32:21.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.411+0000 7f7a77800640 1 -- 172.21.15.102:0/3226604982 shutdown_connections 2024-04-26T20:32:21.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.411+0000 7f7a77800640 1 --2- 172.21.15.102:0/3226604982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a70105d10 0x7f7a70108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.411+0000 7f7a77800640 1 --2- 172.21.15.102:0/3226604982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a70108640 0x7f7a7010aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.411+0000 7f7a77800640 1 --2- 172.21.15.102:0/3226604982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a701033e0 0x7f7a701057d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.411+0000 7f7a77800640 1 -- 172.21.15.102:0/3226604982 >> 172.21.15.102:0/3226604982 conn(0x7f7a700fcdf0 msgr2=0x7f7a700ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:21.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.411+0000 7f7a77800640 1 -- 172.21.15.102:0/3226604982 shutdown_connections 2024-04-26T20:32:21.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.411+0000 7f7a77800640 1 -- 172.21.15.102:0/3226604982 wait complete. 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a77800640 1 Processor -- start 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a77800640 1 -- start start 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a77800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a701033e0 0x7f7a7019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a77800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a70105d10 0x7f7a7019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a77800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a70108640 0x7f7a701a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a77800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010e500 con 0x7f7a701033e0 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a77800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010e380 con 0x7f7a70108640 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a77800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7010e680 con 0x7f7a70105d10 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a74e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a701033e0 0x7f7a7019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a75800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a70108640 0x7f7a701a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.412+0000 7f7a75800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a70108640 0x7f7a701a2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:36480/0 (socket says 172.21.15.102:36480) 2024-04-26T20:32:21.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a74e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a701033e0 0x7f7a7019add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45398/0 (socket says 172.21.15.102:45398) 2024-04-26T20:32:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a75800640 1 -- 172.21.15.102:0/3765383699 learned_addr learned my addr 172.21.15.102:0/3765383699 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a6fe00640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a70105d10 0x7f7a7019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a75800640 1 -- 172.21.15.102:0/3765383699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a70105d10 msgr2=0x7f7a7019b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a75800640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a70105d10 0x7f7a7019b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a75800640 1 -- 172.21.15.102:0/3765383699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a701033e0 msgr2=0x7f7a7019add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a75800640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a701033e0 0x7f7a7019add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a75800640 1 -- 172.21.15.102:0/3765383699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a701a29f0 con 0x7f7a70108640 2024-04-26T20:32:21.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a74e00640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a701033e0 0x7f7a7019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a75800640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a70108640 0x7f7a701a2380 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800eb10 tx=0x7f7a6800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a6d600640 1 -- 172.21.15.102:0/3765383699 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68004b20 con 0x7f7a70108640 2024-04-26T20:32:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a77800640 1 -- 172.21.15.102:0/3765383699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a701a2ce0 con 0x7f7a70108640 2024-04-26T20:32:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a6d600640 1 -- 172.21.15.102:0/3765383699 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a68004cc0 con 0x7f7a70108640 2024-04-26T20:32:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.413+0000 7f7a6d600640 1 -- 172.21.15.102:0/3765383699 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68010690 con 0x7f7a70108640 2024-04-26T20:32:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.414+0000 7f7a77800640 1 -- 172.21.15.102:0/3765383699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a701a31d0 con 0x7f7a70108640 2024-04-26T20:32:21.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.415+0000 7f7a77800640 1 -- 172.21.15.102:0/3765383699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a7010a190 con 0x7f7a70108640 2024-04-26T20:32:21.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.416+0000 7f7a6d600640 1 -- 172.21.15.102:0/3765383699 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f7a68005020 con 0x7f7a70108640 2024-04-26T20:32:21.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.418+0000 7f7a6d600640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7a580a3bb0 0x7f7a580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:21.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.418+0000 7f7a6d600640 1 -- 172.21.15.102:0/3765383699 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7a680d0530 con 0x7f7a70108640 2024-04-26T20:32:21.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.419+0000 7f7a74e00640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7a580a3bb0 0x7f7a580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:21.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.419+0000 7f7a74e00640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7a580a3bb0 0x7f7a580a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7a64031500 tx=0x7f7a64005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:21.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.422+0000 7f7a6d600640 1 -- 172.21.15.102:0/3765383699 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7a6809d310 con 0x7f7a70108640 2024-04-26T20:32:21.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.669+0000 7f7a77800640 1 -- 172.21.15.102:0/3765383699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a7002dad0 con 0x7f7a70108640 2024-04-26T20:32:21.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.670+0000 7f7a6d600640 1 -- 172.21.15.102:0/3765383699 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10500 (secure 0 0 0) 0x7f7a6809cc10 con 0x7f7a70108640 2024-04-26T20:32:21.671 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:21.671 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-26T20:32:21.671 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 13 2024-04-26T20:32:21.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 -- 172.21.15.102:0/3765383699 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7a580a3bb0 msgr2=0x7f7a580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:21.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7a580a3bb0 0x7f7a580a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7a64031500 tx=0x7f7a64005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 -- 172.21.15.102:0/3765383699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a70108640 msgr2=0x7f7a701a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a70108640 0x7f7a701a2380 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800eb10 tx=0x7f7a6800c2d0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 -- 172.21.15.102:0/3765383699 shutdown_connections 2024-04-26T20:32:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7a580a3bb0 0x7f7a580a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a70108640 0x7f7a701a2380 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a70105d10 0x7f7a7019b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 --2- 172.21.15.102:0/3765383699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a701033e0 0x7f7a7019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 -- 172.21.15.102:0/3765383699 >> 172.21.15.102:0/3765383699 conn(0x7f7a700fcdf0 msgr2=0x7f7a70102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.673+0000 7f7a77800640 1 -- 172.21.15.102:0/3765383699 shutdown_connections 2024-04-26T20:32:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.674+0000 7f7a77800640 1 -- 172.21.15.102:0/3765383699 wait complete. 2024-04-26T20:32:21.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:32:21.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.884+0000 7f604f400640 1 -- 172.21.15.102:0/1960318927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6048101b40 msgr2=0x7f6048101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:21.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.884+0000 7f604f400640 1 --2- 172.21.15.102:0/1960318927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6048101b40 0x7f6048101fa0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6034009900 tx=0x7f603402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.884+0000 7f604f400640 1 -- 172.21.15.102:0/1960318927 shutdown_connections 2024-04-26T20:32:21.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.884+0000 7f604f400640 1 --2- 172.21.15.102:0/1960318927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60481024e0 0x7f6048102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.884+0000 7f604f400640 1 --2- 172.21.15.102:0/1960318927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6048101b40 0x7f6048101fa0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.884+0000 7f604f400640 1 --2- 172.21.15.102:0/1960318927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60481011e0 0x7f6048101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.884+0000 7f604f400640 1 -- 172.21.15.102:0/1960318927 >> 172.21.15.102:0/1960318927 conn(0x7f60480fcde0 msgr2=0x7f60480ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:21.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.884+0000 7f604f400640 1 -- 172.21.15.102:0/1960318927 shutdown_connections 2024-04-26T20:32:21.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.884+0000 7f604f400640 1 -- 172.21.15.102:0/1960318927 wait complete. 2024-04-26T20:32:21.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.885+0000 7f604f400640 1 Processor -- start 2024-04-26T20:32:21.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604f400640 1 -- start start 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604f400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60481011e0 0x7f604819ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048101b40 0x7f604819b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604f400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60481024e0 0x7f60481a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604f400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6048110660 con 0x7f6048101b40 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604f400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f60481104e0 con 0x7f60481024e0 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604f400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f60481107e0 con 0x7f60481011e0 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604d400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60481024e0 0x7f60481a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604d400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60481024e0 0x7f60481a22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:36502/0 (socket says 172.21.15.102:36502) 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604d400640 1 -- 172.21.15.102:0/1851951919 learned_addr learned my addr 172.21.15.102:0/1851951919 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f6047e00640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048101b40 0x7f604819b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.886+0000 7f604ca00640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60481011e0 0x7f604819ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f604d400640 1 -- 172.21.15.102:0/1851951919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60481011e0 msgr2=0x7f604819ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:21.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f604d400640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60481011e0 0x7f604819ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f604d400640 1 -- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048101b40 msgr2=0x7f604819b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f604d400640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048101b40 0x7f604819b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f604d400640 1 -- 172.21.15.102:0/1851951919 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60481a2940 con 0x7f60481024e0 2024-04-26T20:32:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f6047e00640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048101b40 0x7f604819b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f604d400640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60481024e0 0x7f60481a22a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f604000d750 tx=0x7f604000dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f6045600640 1 -- 172.21.15.102:0/1851951919 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040017070 con 0x7f60481024e0 2024-04-26T20:32:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f604f400640 1 -- 172.21.15.102:0/1851951919 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60481a2c30 con 0x7f60481024e0 2024-04-26T20:32:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f6045600640 1 -- 172.21.15.102:0/1851951919 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60400054d0 con 0x7f60481024e0 2024-04-26T20:32:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f6045600640 1 -- 172.21.15.102:0/1851951919 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040004c60 con 0x7f60481024e0 2024-04-26T20:32:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.887+0000 7f604f400640 1 -- 172.21.15.102:0/1851951919 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60481a3180 con 0x7f60481024e0 2024-04-26T20:32:21.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.888+0000 7f604f400640 1 -- 172.21.15.102:0/1851951919 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f604810c0b0 con 0x7f60481024e0 2024-04-26T20:32:21.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.890+0000 7f6045600640 1 -- 172.21.15.102:0/1851951919 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f6040009bd0 con 0x7f60481024e0 2024-04-26T20:32:21.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.892+0000 7f6045600640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bb0 0x7f602c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:21.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.892+0000 7f6045600640 1 -- 172.21.15.102:0/1851951919 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f60400cfa20 con 0x7f60481024e0 2024-04-26T20:32:21.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.892+0000 7f604ca00640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bb0 0x7f602c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:21.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.893+0000 7f604ca00640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bb0 0x7f602c0a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f604819c200 tx=0x7f60380075b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:21.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:21.900+0000 7f6045600640 1 -- 172.21.15.102:0/1851951919 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f604009c800 con 0x7f60481024e0 2024-04-26T20:32:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.107+0000 7f604f400640 1 -- 172.21.15.102:0/1851951919 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6048063560 con 0x7f60481024e0 2024-04-26T20:32:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.109+0000 7f6045600640 1 -- 172.21.15.102:0/1851951919 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18492 (secure 0 0 0) 0x7f604009c100 con 0x7f60481024e0 2024-04-26T20:32:22.109 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:22.109 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:31:29.744387+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:32:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 -- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bb0 msgr2=0x7f602c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bb0 0x7f602c0a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f604819c200 tx=0x7f60380075b0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 -- 172.21.15.102:0/1851951919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60481024e0 msgr2=0x7f60481a22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60481024e0 0x7f60481a22a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f604000d750 tx=0x7f604000dc10 comp rx=0 tx=0).stop 2024-04-26T20:32:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 -- 172.21.15.102:0/1851951919 shutdown_connections 2024-04-26T20:32:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bb0 0x7f602c0a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048101b40 0x7f604819b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60481024e0 0x7f60481a22a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 --2- 172.21.15.102:0/1851951919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60481011e0 0x7f604819ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 -- 172.21.15.102:0/1851951919 >> 172.21.15.102:0/1851951919 conn(0x7f60480fcde0 msgr2=0x7f60480fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:22.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 -- 172.21.15.102:0/1851951919 shutdown_connections 2024-04-26T20:32:22.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.112+0000 7f604f400640 1 -- 172.21.15.102:0/1851951919 wait complete. 2024-04-26T20:32:22.132 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:27:10.527933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T20:32:22.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:32:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.319+0000 7f68fe600640 1 -- 172.21.15.102:0/1462745277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f8076970 msgr2=0x7f68f8074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.319+0000 7f68fe600640 1 --2- 172.21.15.102:0/1462745277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f8076970 0x7f68f8074dc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009930 tx=0x7f68ec02a420 comp rx=0 tx=0).stop 2024-04-26T20:32:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.320+0000 7f68fe600640 1 -- 172.21.15.102:0/1462745277 shutdown_connections 2024-04-26T20:32:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.320+0000 7f68fe600640 1 --2- 172.21.15.102:0/1462745277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68f810ee90 0x7f68f8111280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.320+0000 7f68fe600640 1 --2- 172.21.15.102:0/1462745277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f8075390 0x7f68f8075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.320+0000 7f68fe600640 1 --2- 172.21.15.102:0/1462745277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f8076970 0x7f68f8074dc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.320+0000 7f68fe600640 1 -- 172.21.15.102:0/1462745277 >> 172.21.15.102:0/1462745277 conn(0x7f68f80fde30 msgr2=0x7f68f8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.320+0000 7f68fe600640 1 -- 172.21.15.102:0/1462745277 shutdown_connections 2024-04-26T20:32:22.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.320+0000 7f68fe600640 1 -- 172.21.15.102:0/1462745277 wait complete. 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.321+0000 7f68fe600640 1 Processor -- start 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.321+0000 7f68fe600640 1 -- start start 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.321+0000 7f68fe600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f8075390 0x7f68f819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.321+0000 7f68fe600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f8076970 0x7f68f819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.321+0000 7f68fe600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68f810ee90 0x7f68f81a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.321+0000 7f68fe600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f68f81149a0 con 0x7f68f810ee90 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.321+0000 7f68fe600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68f8114820 con 0x7f68f8075390 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.321+0000 7f68fe600640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f68f8114b20 con 0x7f68f8076970 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fd200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f8075390 0x7f68f819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fdc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68f810ee90 0x7f68f81a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fd200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f8075390 0x7f68f819f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:36516/0 (socket says 172.21.15.102:36516) 2024-04-26T20:32:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fd200640 1 -- 172.21.15.102:0/1633794621 learned_addr learned my addr 172.21.15.102:0/1633794621 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68f7e00640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f8076970 0x7f68f819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fd200640 1 -- 172.21.15.102:0/1633794621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f8076970 msgr2=0x7f68f819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fd200640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f8076970 0x7f68f819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fd200640 1 -- 172.21.15.102:0/1633794621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68f810ee90 msgr2=0x7f68f81a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fd200640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68f810ee90 0x7f68f81a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fd200640 1 -- 172.21.15.102:0/1633794621 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f81a6ce0 con 0x7f68f8075390 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fdc00640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68f810ee90 0x7f68f81a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.322+0000 7f68fd200640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f8075390 0x7f68f819f110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f68ec031500 tx=0x7f68ec003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.323+0000 7f68f5600640 1 -- 172.21.15.102:0/1633794621 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec03f070 con 0x7f68f8075390 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.323+0000 7f68f5600640 1 -- 172.21.15.102:0/1633794621 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68ec031c50 con 0x7f68f8075390 2024-04-26T20:32:22.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.323+0000 7f68fe600640 1 -- 172.21.15.102:0/1633794621 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f81a6f70 con 0x7f68f8075390 2024-04-26T20:32:22.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.323+0000 7f68f5600640 1 -- 172.21.15.102:0/1633794621 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec035030 con 0x7f68f8075390 2024-04-26T20:32:22.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.323+0000 7f68fe600640 1 -- 172.21.15.102:0/1633794621 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f81a7380 con 0x7f68f8075390 2024-04-26T20:32:22.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.325+0000 7f68fe600640 1 -- 172.21.15.102:0/1633794621 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f8105060 con 0x7f68f8075390 2024-04-26T20:32:22.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.326+0000 7f68f5600640 1 -- 172.21.15.102:0/1633794621 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f68ec042070 con 0x7f68f8075390 2024-04-26T20:32:22.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.328+0000 7f68f5600640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68e00a3bb0 0x7f68e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:22.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.328+0000 7f68f5600640 1 -- 172.21.15.102:0/1633794621 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f68ec0f06d0 con 0x7f68f8075390 2024-04-26T20:32:22.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.328+0000 7f68f7e00640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68e00a3bb0 0x7f68e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:22.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.329+0000 7f68f7e00640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68e00a3bb0 0x7f68e00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f68f81a05d0 tx=0x7f68e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:22.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.331+0000 7f68f5600640 1 -- 172.21.15.102:0/1633794621 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f68ec0bce60 con 0x7f68f8075390 2024-04-26T20:32:22.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:22 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3765383699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:22.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:22 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1851951919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:22.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:22 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3765383699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:22.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:22 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1851951919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:22.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:22 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3765383699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:22.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:22 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1851951919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:22.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.546+0000 7f68fe600640 1 -- 172.21.15.102:0/1633794621 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f68f8063530 con 0x7f68f8075390 2024-04-26T20:32:22.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.547+0000 7f68f5600640 1 -- 172.21.15.102:0/1633794621 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f68ec0c1160 con 0x7f68f8075390 2024-04-26T20:32:22.548 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.549+0000 7f68fe600640 1 -- 172.21.15.102:0/1633794621 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68e00a3bb0 msgr2=0x7f68e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.549+0000 7f68fe600640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68e00a3bb0 0x7f68e00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f68f81a05d0 tx=0x7f68e4005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.549+0000 7f68fe600640 1 -- 172.21.15.102:0/1633794621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f8075390 msgr2=0x7f68f819f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.549+0000 7f68fe600640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f8075390 0x7f68f819f110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f68ec031500 tx=0x7f68ec003c10 comp rx=0 tx=0).stop 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.549+0000 7f68fe600640 1 -- 172.21.15.102:0/1633794621 shutdown_connections 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.549+0000 7f68fe600640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68e00a3bb0 0x7f68e00a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.549+0000 7f68fe600640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68f810ee90 0x7f68f81a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.549+0000 7f68fe600640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f8076970 0x7f68f819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.549+0000 7f68fe600640 1 --2- 172.21.15.102:0/1633794621 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68f8075390 0x7f68f819f110 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.550+0000 7f68fe600640 1 -- 172.21.15.102:0/1633794621 >> 172.21.15.102:0/1633794621 conn(0x7f68f80fde30 msgr2=0x7f68f8102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:22.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.550+0000 7f68fe600640 1 -- 172.21.15.102:0/1633794621 shutdown_connections 2024-04-26T20:32:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.550+0000 7f68fe600640 1 -- 172.21.15.102:0/1633794621 wait complete. 2024-04-26T20:32:22.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T20:32:22.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.762+0000 7f705c000640 1 -- 172.21.15.102:0/1374008381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7054103490 msgr2=0x7f7054105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.762+0000 7f705c000640 1 --2- 172.21.15.102:0/1374008381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7054103490 0x7f7054105880 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7048009900 tx=0x7f704802a410 comp rx=0 tx=0).stop 2024-04-26T20:32:22.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.762+0000 7f705c000640 1 -- 172.21.15.102:0/1374008381 shutdown_connections 2024-04-26T20:32:22.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.762+0000 7f705c000640 1 --2- 172.21.15.102:0/1374008381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70541086f0 0x7f705410aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.762+0000 7f705c000640 1 --2- 172.21.15.102:0/1374008381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7054105dc0 0x7f70541081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.762+0000 7f705c000640 1 --2- 172.21.15.102:0/1374008381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7054103490 0x7f7054105880 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.762+0000 7f705c000640 1 -- 172.21.15.102:0/1374008381 >> 172.21.15.102:0/1374008381 conn(0x7f70540fcde0 msgr2=0x7f70540ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:22.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.762+0000 7f705c000640 1 -- 172.21.15.102:0/1374008381 shutdown_connections 2024-04-26T20:32:22.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.762+0000 7f705c000640 1 -- 172.21.15.102:0/1374008381 wait complete. 2024-04-26T20:32:22.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.763+0000 7f705c000640 1 Processor -- start 2024-04-26T20:32:22.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.763+0000 7f705c000640 1 -- start start 2024-04-26T20:32:22.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f705c000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7054103490 0x7f7054102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:22.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f705c000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7054105dc0 0x7f7054101440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:22.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f705c000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70541086f0 0x7f7054101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f705c000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f705410e4e0 con 0x7f7054105dc0 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f705c000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f705410e360 con 0x7f7054103490 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f705c000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f705410e660 con 0x7f70541086f0 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f7058c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7054105dc0 0x7f7054101440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f7058c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7054105dc0 0x7f7054101440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45442/0 (socket says 172.21.15.102:45442) 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f7059600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7054103490 0x7f7054102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f7058c00640 1 -- 172.21.15.102:0/2437273077 learned_addr learned my addr 172.21.15.102:0/2437273077 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f705a000640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70541086f0 0x7f7054101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.764+0000 7f7058c00640 1 -- 172.21.15.102:0/2437273077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70541086f0 msgr2=0x7f7054101980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f7058c00640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70541086f0 0x7f7054101980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f7058c00640 1 -- 172.21.15.102:0/2437273077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7054103490 msgr2=0x7f7054102e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f7058c00640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7054103490 0x7f7054102e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f7058c00640 1 -- 172.21.15.102:0/2437273077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7054102210 con 0x7f7054105dc0 2024-04-26T20:32:22.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f7059600640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7054103490 0x7f7054102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:22.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f705a000640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70541086f0 0x7f7054101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:22.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f7058c00640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7054105dc0 0x7f7054101440 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f704000e9a0 tx=0x7f704000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:22.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f7052000640 1 -- 172.21.15.102:0/2437273077 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704000cd80 con 0x7f7054105dc0 2024-04-26T20:32:22.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f705c000640 1 -- 172.21.15.102:0/2437273077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70541a2d00 con 0x7f7054105dc0 2024-04-26T20:32:22.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f7052000640 1 -- 172.21.15.102:0/2437273077 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70400035f0 con 0x7f7054105dc0 2024-04-26T20:32:22.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f7052000640 1 -- 172.21.15.102:0/2437273077 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040010690 con 0x7f7054105dc0 2024-04-26T20:32:22.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.765+0000 7f705c000640 1 -- 172.21.15.102:0/2437273077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70541a31a0 con 0x7f7054105dc0 2024-04-26T20:32:22.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.767+0000 7f705c000640 1 -- 172.21.15.102:0/2437273077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70540fe8d0 con 0x7f7054105dc0 2024-04-26T20:32:22.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.769+0000 7f7052000640 1 -- 172.21.15.102:0/2437273077 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f7040003fd0 con 0x7f7054105dc0 2024-04-26T20:32:22.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.770+0000 7f7052000640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f703c0a3bb0 0x7f703c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:22.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.770+0000 7f7052000640 1 -- 172.21.15.102:0/2437273077 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f70400d0280 con 0x7f7054105dc0 2024-04-26T20:32:22.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.771+0000 7f7059600640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f703c0a3bb0 0x7f703c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:22.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.771+0000 7f7059600640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f703c0a3bb0 0x7f703c0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7048031500 tx=0x7f7048005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:22.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.779+0000 7f7052000640 1 -- 172.21.15.102:0/2437273077 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f704009d060 con 0x7f7054105dc0 2024-04-26T20:32:22.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.993+0000 7f705c000640 1 -- 172.21.15.102:0/2437273077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7054063530 con 0x7f7054105dc0 2024-04-26T20:32:22.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.994+0000 7f7052000640 1 -- 172.21.15.102:0/2437273077 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f704009c960 con 0x7f7054105dc0 2024-04-26T20:32:22.995 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:22.995 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-04-26T20:32:22.995 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-26T20:32:22.995 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2024-04-26T20:32:22.995 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2024-04-26T20:32:22.995 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-26T20:32:22.996 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-04-26T20:32:22.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 -- 172.21.15.102:0/2437273077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f703c0a3bb0 msgr2=0x7f703c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f703c0a3bb0 0x7f703c0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7048031500 tx=0x7f7048005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 -- 172.21.15.102:0/2437273077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7054105dc0 msgr2=0x7f7054101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7054105dc0 0x7f7054101440 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f704000e9a0 tx=0x7f704000ee60 comp rx=0 tx=0).stop 2024-04-26T20:32:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 -- 172.21.15.102:0/2437273077 shutdown_connections 2024-04-26T20:32:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f703c0a3bb0 0x7f703c0a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7054105dc0 0x7f7054101440 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70541086f0 0x7f7054101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 --2- 172.21.15.102:0/2437273077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7054103490 0x7f7054102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 -- 172.21.15.102:0/2437273077 >> 172.21.15.102:0/2437273077 conn(0x7f70540fcde0 msgr2=0x7f7054103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.996+0000 7f705c000640 1 -- 172.21.15.102:0/2437273077 shutdown_connections 2024-04-26T20:32:22.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:22.997+0000 7f705c000640 1 -- 172.21.15.102:0/2437273077 wait complete. 2024-04-26T20:32:23.016 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T20:32:23.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T20:32:23.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.202+0000 7f8262a00640 1 -- 172.21.15.102:0/3779866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825c106740 msgr2=0x7f825c10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:23.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.202+0000 7f8262a00640 1 --2- 172.21.15.102:0/3779866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825c106740 0x7f825c10cfd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8254009900 tx=0x7f825402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:23.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.202+0000 7f8262a00640 1 -- 172.21.15.102:0/3779866 shutdown_connections 2024-04-26T20:32:23.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.202+0000 7f8262a00640 1 --2- 172.21.15.102:0/3779866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825c106740 0x7f825c10cfd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:23.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.202+0000 7f8262a00640 1 --2- 172.21.15.102:0/3779866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f825c105d80 0x7f825c106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:23.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.202+0000 7f8262a00640 1 --2- 172.21.15.102:0/3779866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f825c104b90 0x7f825c104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:23.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.202+0000 7f8262a00640 1 -- 172.21.15.102:0/3779866 >> 172.21.15.102:0/3779866 conn(0x7f825c1000e0 msgr2=0x7f825c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:23.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.202+0000 7f8262a00640 1 -- 172.21.15.102:0/3779866 shutdown_connections 2024-04-26T20:32:23.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.202+0000 7f8262a00640 1 -- 172.21.15.102:0/3779866 wait complete. 2024-04-26T20:32:23.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.203+0000 7f8262a00640 1 Processor -- start 2024-04-26T20:32:23.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f8262a00640 1 -- start start 2024-04-26T20:32:23.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f8262a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f825c104b90 0x7f825c19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:23.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f8262a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825c105d80 0x7f825c19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:23.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f8262a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f825c106740 0x7f825c1a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f8262a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f825c110710 con 0x7f825c105d80 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f8262a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f825c110590 con 0x7f825c106740 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f8262a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f825c110890 con 0x7f825c104b90 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f825be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f825c104b90 0x7f825c19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f825be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f825c104b90 0x7f825c19ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:44534/0 (socket says 172.21.15.102:44534) 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f825be00640 1 -- 172.21.15.102:0/2766623674 learned_addr learned my addr 172.21.15.102:0/2766623674 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f8253400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825c105d80 0x7f825c19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.204+0000 7f8260a00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f825c106740 0x7f825c1a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f825be00640 1 -- 172.21.15.102:0/2766623674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f825c106740 msgr2=0x7f825c1a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f825be00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f825c106740 0x7f825c1a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:23.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f825be00640 1 -- 172.21.15.102:0/2766623674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825c105d80 msgr2=0x7f825c19b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:23.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f825be00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825c105d80 0x7f825c19b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:23.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f825be00640 1 -- 172.21.15.102:0/2766623674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c1a29b0 con 0x7f825c104b90 2024-04-26T20:32:23.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f8260a00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f825c106740 0x7f825c1a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:23.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f825be00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f825c104b90 0x7f825c19ada0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f824c009d20 tx=0x7f824c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:23.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f8259600640 1 -- 172.21.15.102:0/2766623674 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c010630 con 0x7f825c104b90 2024-04-26T20:32:23.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f8259600640 1 -- 172.21.15.102:0/2766623674 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f824c010c80 con 0x7f825c104b90 2024-04-26T20:32:23.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f8259600640 1 -- 172.21.15.102:0/2766623674 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c018d50 con 0x7f825c104b90 2024-04-26T20:32:23.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f8262a00640 1 -- 172.21.15.102:0/2766623674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c1a2ca0 con 0x7f825c104b90 2024-04-26T20:32:23.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.205+0000 7f8262a00640 1 -- 172.21.15.102:0/2766623674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c1a31a0 con 0x7f825c104b90 2024-04-26T20:32:23.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.206+0000 7f8262a00640 1 -- 172.21.15.102:0/2766623674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f825c10beb0 con 0x7f825c104b90 2024-04-26T20:32:23.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.208+0000 7f8259600640 1 -- 172.21.15.102:0/2766623674 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f824c01f020 con 0x7f825c104b90 2024-04-26T20:32:23.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.209+0000 7f8259600640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82400a3bb0 0x7f82400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:23.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.210+0000 7f8259600640 1 -- 172.21.15.102:0/2766623674 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f824c0d04c0 con 0x7f825c104b90 2024-04-26T20:32:23.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.210+0000 7f8253400640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82400a3bb0 0x7f82400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:23.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.210+0000 7f8253400640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82400a3bb0 0x7f82400a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8244009bd0 tx=0x7f8244005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.214+0000 7f8259600640 1 -- 172.21.15.102:0/2766623674 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f824c09d2a0 con 0x7f825c104b90 2024-04-26T20:32:23.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:23 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:23 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:23 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:23 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:23 smithi102 ceph-mon[28094]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:23.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:23 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1633794621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:32:23.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:23 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2437273077' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T20:32:23.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:23.422+0000 7f8262a00640 1 -- 172.21.15.102:0/2766623674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f825c02dad0 con 0x7f825c104b90 2024-04-26T20:32:23.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:23 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:23 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:23 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:23 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:23 smithi112 ceph-mon[32769]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:23.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:23 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1633794621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:32:23.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:23 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2437273077' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T20:32:23.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:23 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:23 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:23 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:23 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:23.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:23 smithi130 ceph-mon[32704]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:23.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:23 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1633794621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:32:23.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:23 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2437273077' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T20:32:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.239+0000 7f8259600640 1 -- 172.21.15.102:0/2766623674 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f824c09cba0 con 0x7f825c104b90 2024-04-26T20:32:24.241 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2024-04-26T20:32:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 -- 172.21.15.102:0/2766623674 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82400a3bb0 msgr2=0x7f82400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82400a3bb0 0x7f82400a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8244009bd0 tx=0x7f8244005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 -- 172.21.15.102:0/2766623674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f825c104b90 msgr2=0x7f825c19ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f825c104b90 0x7f825c19ada0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f824c009d20 tx=0x7f824c00c6a0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 -- 172.21.15.102:0/2766623674 shutdown_connections 2024-04-26T20:32:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82400a3bb0 0x7f82400a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825c105d80 0x7f825c19b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f825c106740 0x7f825c1a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 --2- 172.21.15.102:0/2766623674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f825c104b90 0x7f825c19ada0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f824c009d20 tx=0x7f824c00c6a0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 -- 172.21.15.102:0/2766623674 >> 172.21.15.102:0/2766623674 conn(0x7f825c1000e0 msgr2=0x7f825c109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 -- 172.21.15.102:0/2766623674 shutdown_connections 2024-04-26T20:32:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.244+0000 7f8262a00640 1 -- 172.21.15.102:0/2766623674 wait complete. 2024-04-26T20:32:24.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-26T20:32:24.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.454+0000 7ff6a8e00640 1 -- 172.21.15.102:0/3879201713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6a40772f0 msgr2=0x7ff6a4077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:24.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.454+0000 7ff6a8e00640 1 --2- 172.21.15.102:0/3879201713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6a40772f0 0x7ff6a4077710 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff694009900 tx=0x7ff69402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.454+0000 7ff6a8e00640 1 -- 172.21.15.102:0/3879201713 shutdown_connections 2024-04-26T20:32:24.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.454+0000 7ff6a8e00640 1 --2- 172.21.15.102:0/3879201713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a4078ed0 0x7ff6a4079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.454+0000 7ff6a8e00640 1 --2- 172.21.15.102:0/3879201713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6a4078530 0x7ff6a4078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.454+0000 7ff6a8e00640 1 --2- 172.21.15.102:0/3879201713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6a40772f0 0x7ff6a4077710 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.454+0000 7ff6a8e00640 1 -- 172.21.15.102:0/3879201713 >> 172.21.15.102:0/3879201713 conn(0x7ff6a41000e0 msgr2=0x7ff6a4102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:24.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.454+0000 7ff6a8e00640 1 -- 172.21.15.102:0/3879201713 shutdown_connections 2024-04-26T20:32:24.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.454+0000 7ff6a8e00640 1 -- 172.21.15.102:0/3879201713 wait complete. 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.455+0000 7ff6a8e00640 1 Processor -- start 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a8e00640 1 -- start start 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a40772f0 0x7ff6a419f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a8e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6a4078530 0x7ff6a419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a8e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6a4078ed0 0x7ff6a41a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a8e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a4114ad0 con 0x7ff6a40772f0 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a8e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a4114950 con 0x7ff6a4078ed0 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a8e00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a4114c50 con 0x7ff6a4078530 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a40772f0 0x7ff6a419f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a2800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6a4078ed0 0x7ff6a41a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a40772f0 0x7ff6a419f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45476/0 (socket says 172.21.15.102:45476) 2024-04-26T20:32:24.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a1e00640 1 -- 172.21.15.102:0/2552216728 learned_addr learned my addr 172.21.15.102:0/2552216728 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.456+0000 7ff6a1400640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6a4078530 0x7ff6a419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff6a2800640 1 -- 172.21.15.102:0/2552216728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6a4078530 msgr2=0x7ff6a419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff6a2800640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6a4078530 0x7ff6a419f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff6a2800640 1 -- 172.21.15.102:0/2552216728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a40772f0 msgr2=0x7ff6a419f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff6a2800640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a40772f0 0x7ff6a419f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff6a2800640 1 -- 172.21.15.102:0/2552216728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a41a6e20 con 0x7ff6a4078ed0 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff6a2800640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6a4078ed0 0x7ff6a41a67b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00da10 tx=0x7ff69c00ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff69aa00640 1 -- 172.21.15.102:0/2552216728 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c00bd70 con 0x7ff6a4078ed0 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff69aa00640 1 -- 172.21.15.102:0/2552216728 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff69c003e20 con 0x7ff6a4078ed0 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff6a8e00640 1 -- 172.21.15.102:0/2552216728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a41a7110 con 0x7ff6a4078ed0 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff69aa00640 1 -- 172.21.15.102:0/2552216728 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c0106c0 con 0x7ff6a4078ed0 2024-04-26T20:32:24.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.457+0000 7ff6a8e00640 1 -- 172.21.15.102:0/2552216728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a41a7660 con 0x7ff6a4078ed0 2024-04-26T20:32:24.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.458+0000 7ff6a8e00640 1 -- 172.21.15.102:0/2552216728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a41117e0 con 0x7ff6a4078ed0 2024-04-26T20:32:24.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.460+0000 7ff69aa00640 1 -- 172.21.15.102:0/2552216728 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ff69c004270 con 0x7ff6a4078ed0 2024-04-26T20:32:24.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.462+0000 7ff69aa00640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff6880a3c30 0x7ff6880a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:24.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.462+0000 7ff69aa00640 1 -- 172.21.15.102:0/2552216728 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7ff69c0d04b0 con 0x7ff6a4078ed0 2024-04-26T20:32:24.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.462+0000 7ff6a1e00640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff6880a3c30 0x7ff6880a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:24.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.463+0000 7ff6a1e00640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff6880a3c30 0x7ff6880a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff694009230 tx=0x7ff6940058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:24.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.466+0000 7ff69aa00640 1 -- 172.21.15.102:0/2552216728 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff69c09d120 con 0x7ff6a4078ed0 2024-04-26T20:32:24.478 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.478 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.478 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:24.478 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:32:24.478 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.478 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:24 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2766623674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:32:24.479 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:24 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:32:24.479 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.479 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:32:24.488 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.488 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.488 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:24.488 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:32:24.488 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.489 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:24 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2766623674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:32:24.489 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:24 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:32:24.489 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.489 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:32:24.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:24.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:32:24.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:24 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2766623674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:32:24.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:24 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T20:32:24.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:24.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:32:24.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:24.684+0000 7ff6a8e00640 1 -- 172.21.15.102:0/2552216728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7ff6a4063530 con 0x7ff6a4078ed0 2024-04-26T20:32:25.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.256+0000 7ff69aa00640 1 -- 172.21.15.102:0/2552216728 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7ff69c09ca20 con 0x7ff6a4078ed0 2024-04-26T20:32:25.258 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2024-04-26T20:32:25.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.262+0000 7ff6a8e00640 1 -- 172.21.15.102:0/2552216728 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff6880a3c30 msgr2=0x7ff6880a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:25.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.262+0000 7ff6a8e00640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff6880a3c30 0x7ff6880a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff694009230 tx=0x7ff6940058d0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.262+0000 7ff6a8e00640 1 -- 172.21.15.102:0/2552216728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6a4078ed0 msgr2=0x7ff6a41a67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:25.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.262+0000 7ff6a8e00640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6a4078ed0 0x7ff6a41a67b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00da10 tx=0x7ff69c00ded0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.266+0000 7ff6a8e00640 1 -- 172.21.15.102:0/2552216728 shutdown_connections 2024-04-26T20:32:25.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.266+0000 7ff6a8e00640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff6880a3c30 0x7ff6880a60f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.266+0000 7ff6a8e00640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6a4078ed0 0x7ff6a41a67b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.266+0000 7ff6a8e00640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6a4078530 0x7ff6a419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.266+0000 7ff6a8e00640 1 --2- 172.21.15.102:0/2552216728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a40772f0 0x7ff6a419f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.266+0000 7ff6a8e00640 1 -- 172.21.15.102:0/2552216728 >> 172.21.15.102:0/2552216728 conn(0x7ff6a41000e0 msgr2=0x7ff6a4101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:25.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.268+0000 7ff6a8e00640 1 -- 172.21.15.102:0/2552216728 shutdown_connections 2024-04-26T20:32:25.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.268+0000 7ff6a8e00640 1 -- 172.21.15.102:0/2552216728 wait complete. 2024-04-26T20:32:25.308 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T20:32:25.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.496+0000 7fe2fa000640 1 -- 172.21.15.102:0/2714952262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2f4103b70 msgr2=0x7fe2f4103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:25.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.496+0000 7fe2fa000640 1 --2- 172.21.15.102:0/2714952262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2f4103b70 0x7fe2f4103ff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009900 tx=0x7fe2e002a410 comp rx=0 tx=0).stop 2024-04-26T20:32:25.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.496+0000 7fe2fa000640 1 -- 172.21.15.102:0/2714952262 shutdown_connections 2024-04-26T20:32:25.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.496+0000 7fe2fa000640 1 --2- 172.21.15.102:0/2714952262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2f4104530 0x7fe2f410adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.496+0000 7fe2fa000640 1 --2- 172.21.15.102:0/2714952262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2f4103b70 0x7fe2f4103ff0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.496+0000 7fe2fa000640 1 --2- 172.21.15.102:0/2714952262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f4102980 0x7fe2f4102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.496+0000 7fe2fa000640 1 -- 172.21.15.102:0/2714952262 >> 172.21.15.102:0/2714952262 conn(0x7fe2f40fdf10 msgr2=0x7fe2f4100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:25.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.496+0000 7fe2fa000640 1 -- 172.21.15.102:0/2714952262 shutdown_connections 2024-04-26T20:32:25.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.496+0000 7fe2fa000640 1 -- 172.21.15.102:0/2714952262 wait complete. 2024-04-26T20:32:25.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2fa000640 1 Processor -- start 2024-04-26T20:32:25.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2fa000640 1 -- start start 2024-04-26T20:32:25.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2fa000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f4102980 0x7fe2f4108700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2fa000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2f4103b70 0x7fe2f4108c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2fa000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2f4104530 0x7fe2f4105270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2fa000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f410e500 con 0x7fe2f4103b70 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2fa000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f410e380 con 0x7fe2f4104530 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2fa000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f410e680 con 0x7fe2f4102980 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2f2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2f4103b70 0x7fe2f4108c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2f2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2f4103b70 0x7fe2f4108c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44704/0 (socket says 172.21.15.102:44704) 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2f2a00640 1 -- 172.21.15.102:0/2494159711 learned_addr learned my addr 172.21.15.102:0/2494159711 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.498+0000 7fe2f3400640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f4102980 0x7fe2f4108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.499+0000 7fe2f3e00640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2f4104530 0x7fe2f4105270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:25.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.499+0000 7fe2f3400640 1 -- 172.21.15.102:0/2494159711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2f4104530 msgr2=0x7fe2f4105270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.499+0000 7fe2f3400640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2f4104530 0x7fe2f4105270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.499+0000 7fe2f3400640 1 -- 172.21.15.102:0/2494159711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2f4103b70 msgr2=0x7fe2f4108c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.499+0000 7fe2f3400640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2f4103b70 0x7fe2f4108c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.499+0000 7fe2f3400640 1 -- 172.21.15.102:0/2494159711 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f4105b30 con 0x7fe2f4102980 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.499+0000 7fe2f3e00640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2f4104530 0x7fe2f4105270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.499+0000 7fe2f3400640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f4102980 0x7fe2f4108700 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400e9c0 tx=0x7fe2e400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.499+0000 7fe2ebe00640 1 -- 172.21.15.102:0/2494159711 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400ccc0 con 0x7fe2f4102980 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.500+0000 7fe2ebe00640 1 -- 172.21.15.102:0/2494159711 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e40035f0 con 0x7fe2f4102980 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.500+0000 7fe2ebe00640 1 -- 172.21.15.102:0/2494159711 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400ce60 con 0x7fe2f4102980 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.500+0000 7fe2fa000640 1 -- 172.21.15.102:0/2494159711 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f41a7230 con 0x7fe2f4102980 2024-04-26T20:32:25.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.500+0000 7fe2fa000640 1 -- 172.21.15.102:0/2494159711 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f41a7640 con 0x7fe2f4102980 2024-04-26T20:32:25.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.501+0000 7fe2fa000640 1 -- 172.21.15.102:0/2494159711 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f41a78d0 con 0x7fe2f4102980 2024-04-26T20:32:25.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.503+0000 7fe2ebe00640 1 -- 172.21.15.102:0/2494159711 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe2e4004000 con 0x7fe2f4102980 2024-04-26T20:32:25.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.504+0000 7fe2ebe00640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2d80a3c30 0x7fe2d80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:25.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.504+0000 7fe2ebe00640 1 -- 172.21.15.102:0/2494159711 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7fe2e40d0670 con 0x7fe2f4102980 2024-04-26T20:32:25.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.504+0000 7fe2f2a00640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2d80a3c30 0x7fe2d80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:25.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.505+0000 7fe2f2a00640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2d80a3c30 0x7fe2d80a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4106680 tx=0x7fe2e0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:25.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.508+0000 7fe2ebe00640 1 -- 172.21.15.102:0/2494159711 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe2e409d160 con 0x7fe2f4102980 2024-04-26T20:32:25.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:25 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T20:32:25.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:25 smithi112 ceph-mon[32769]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T20:32:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:25 smithi112 ceph-mon[32769]: pgmap v310: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:25 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2552216728' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T20:32:25.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:25 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T20:32:25.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:25 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T20:32:25.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:25 smithi130 ceph-mon[32704]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T20:32:25.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:25 smithi130 ceph-mon[32704]: pgmap v310: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:25.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:25 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2552216728' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T20:32:25.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:25 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T20:32:25.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:25 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T20:32:25.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:25 smithi102 ceph-mon[28094]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T20:32:25.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:25 smithi102 ceph-mon[28094]: pgmap v310: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:25.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:25 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2552216728' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T20:32:25.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:25 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T20:32:25.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:25.767+0000 7fe2fa000640 1 -- 172.21.15.102:0/2494159711 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe2f41a7aa0 con 0x7fe2f4102980 2024-04-26T20:32:26.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.305+0000 7fe2ebe00640 1 -- 172.21.15.102:0/2494159711 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fe2e409ca60 con 0x7fe2f4102980 2024-04-26T20:32:26.306 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.307+0000 7fe2fa000640 1 -- 172.21.15.102:0/2494159711 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2d80a3c30 msgr2=0x7fe2d80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2d80a3c30 0x7fe2d80a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4106680 tx=0x7fe2e0005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 -- 172.21.15.102:0/2494159711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f4102980 msgr2=0x7fe2f4108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f4102980 0x7fe2f4108700 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400e9c0 tx=0x7fe2e400ee80 comp rx=0 tx=0).stop 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 -- 172.21.15.102:0/2494159711 shutdown_connections 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2d80a3c30 0x7fe2d80a60f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2f4103b70 0x7fe2f4108c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2f4104530 0x7fe2f4105270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 --2- 172.21.15.102:0/2494159711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f4102980 0x7fe2f4108700 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 -- 172.21.15.102:0/2494159711 >> 172.21.15.102:0/2494159711 conn(0x7fe2f40fdf10 msgr2=0x7fe2f4109340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 -- 172.21.15.102:0/2494159711 shutdown_connections 2024-04-26T20:32:26.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.308+0000 7fe2fa000640 1 -- 172.21.15.102:0/2494159711 wait complete. 2024-04-26T20:32:26.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:32:26.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T20:32:26.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T20:32:26.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2494159711' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:32:26.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:32:26.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T20:32:26.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T20:32:26.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:26.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:32:26.329 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.331 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-26T20:32:26.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-26T20:32:26.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.507+0000 7f10fb600640 1 -- 172.21.15.102:0/2328191146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10f4104b90 msgr2=0x7f10f4104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:26.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.507+0000 7f10fb600640 1 --2- 172.21.15.102:0/2328191146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10f4104b90 0x7f10f4104fb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009900 tx=0x7f10e802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:26.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.507+0000 7f10fb600640 1 -- 172.21.15.102:0/2328191146 shutdown_connections 2024-04-26T20:32:26.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.507+0000 7f10fb600640 1 --2- 172.21.15.102:0/2328191146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10f4106740 0x7f10f410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:26.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.507+0000 7f10fb600640 1 --2- 172.21.15.102:0/2328191146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10f4105d80 0x7f10f4106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:26.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.507+0000 7f10fb600640 1 --2- 172.21.15.102:0/2328191146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10f4104b90 0x7f10f4104fb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:26.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.507+0000 7f10fb600640 1 -- 172.21.15.102:0/2328191146 >> 172.21.15.102:0/2328191146 conn(0x7f10f4100120 msgr2=0x7f10f4102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:26.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.507+0000 7f10fb600640 1 -- 172.21.15.102:0/2328191146 shutdown_connections 2024-04-26T20:32:26.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.507+0000 7f10fb600640 1 -- 172.21.15.102:0/2328191146 wait complete. 2024-04-26T20:32:26.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.508+0000 7f10fb600640 1 Processor -- start 2024-04-26T20:32:26.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.508+0000 7f10fb600640 1 -- start start 2024-04-26T20:32:26.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.508+0000 7f10fb600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10f4104b90 0x7f10f419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:26.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.508+0000 7f10fb600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10f4105d80 0x7f10f419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:26.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.508+0000 7f10fb600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10f4106740 0x7f10f41a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:26.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.508+0000 7f10fb600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f10f41106b0 con 0x7f10f4105d80 2024-04-26T20:32:26.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.508+0000 7f10fb600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f10f4110530 con 0x7f10f4104b90 2024-04-26T20:32:26.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.508+0000 7f10fb600640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f10f4110830 con 0x7f10f4106740 2024-04-26T20:32:26.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.508+0000 7f10f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10f4105d80 0x7f10f419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:26.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f8c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10f4104b90 0x7f10f419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:26.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f8c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10f4104b90 0x7f10f419ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:54956/0 (socket says 172.21.15.102:54956) 2024-04-26T20:32:26.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f8c00640 1 -- 172.21.15.102:0/2053762866 learned_addr learned my addr 172.21.15.102:0/2053762866 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:26.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10f4105d80 0x7f10f419b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44722/0 (socket says 172.21.15.102:44722) 2024-04-26T20:32:26.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f9600640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10f4106740 0x7f10f41a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:26.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f8c00640 1 -- 172.21.15.102:0/2053762866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10f4106740 msgr2=0x7f10f41a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:26.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f8c00640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10f4106740 0x7f10f41a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:26.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f8c00640 1 -- 172.21.15.102:0/2053762866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10f4105d80 msgr2=0x7f10f419b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:26.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f8c00640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10f4105d80 0x7f10f419b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:26.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f8c00640 1 -- 172.21.15.102:0/2053762866 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f41a2a20 con 0x7f10f4104b90 2024-04-26T20:32:26.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f9600640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10f4106740 0x7f10f41a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T20:32:26.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f3e00640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10f4105d80 0x7f10f419b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:26.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f8c00640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10f4104b90 0x7f10f419ae00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f10e8006010 tx=0x7f10e80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:26.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f1600640 1 -- 172.21.15.102:0/2053762866 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e803f030 con 0x7f10f4104b90 2024-04-26T20:32:26.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10fb600640 1 -- 172.21.15.102:0/2053762866 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f41a2cb0 con 0x7f10f4104b90 2024-04-26T20:32:26.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f1600640 1 -- 172.21.15.102:0/2053762866 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10e8031910 con 0x7f10f4104b90 2024-04-26T20:32:26.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.509+0000 7f10f1600640 1 -- 172.21.15.102:0/2053762866 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e8035070 con 0x7f10f4104b90 2024-04-26T20:32:26.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.510+0000 7f10fb600640 1 -- 172.21.15.102:0/2053762866 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f41a31a0 con 0x7f10f4104b90 2024-04-26T20:32:26.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.510+0000 7f10fb600640 1 -- 172.21.15.102:0/2053762866 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10f410bfa0 con 0x7f10f4104b90 2024-04-26T20:32:26.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.513+0000 7f10f1600640 1 -- 172.21.15.102:0/2053762866 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f10e8042030 con 0x7f10f4104b90 2024-04-26T20:32:26.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.514+0000 7f10f1600640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10dc0a3bb0 0x7f10dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:26.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.514+0000 7f10f1600640 1 -- 172.21.15.102:0/2053762866 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f10e80f08f0 con 0x7f10f4104b90 2024-04-26T20:32:26.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.514+0000 7f10f3e00640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10dc0a3bb0 0x7f10dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:26.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.515+0000 7f10f3e00640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10dc0a3bb0 0x7f10dc0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f10f419c310 tx=0x7f10e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:26.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.518+0000 7f10f1600640 1 -- 172.21.15.102:0/2053762866 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f10e80bd380 con 0x7f10f4104b90 2024-04-26T20:32:26.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:32:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T20:32:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T20:32:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2494159711' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:32:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:32:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T20:32:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T20:32:26.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:26.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:32:26.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:32:26.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2494159711' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:32:26.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:32:26.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:32:26.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:26.727+0000 7f10fb600640 1 -- 172.21.15.102:0/2053762866 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f10f4063530 con 0x7f10f4104b90 2024-04-26T20:32:27.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.209+0000 7f10f1600640 1 -- 172.21.15.102:0/2053762866 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f10e8003e60 con 0x7f10f4104b90 2024-04-26T20:32:27.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 -- 172.21.15.102:0/2053762866 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10dc0a3bb0 msgr2=0x7f10dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10dc0a3bb0 0x7f10dc0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f10f419c310 tx=0x7f10e0005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 -- 172.21.15.102:0/2053762866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10f4104b90 msgr2=0x7f10f419ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10f4104b90 0x7f10f419ae00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f10e8006010 tx=0x7f10e80042f0 comp rx=0 tx=0).stop 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 -- 172.21.15.102:0/2053762866 shutdown_connections 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10dc0a3bb0 0x7f10dc0a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10f4105d80 0x7f10f419b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10f4106740 0x7f10f41a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 --2- 172.21.15.102:0/2053762866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10f4104b90 0x7f10f419ae00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 -- 172.21.15.102:0/2053762866 >> 172.21.15.102:0/2053762866 conn(0x7f10f4100120 msgr2=0x7f10f4109170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.213+0000 7f10fb600640 1 -- 172.21.15.102:0/2053762866 shutdown_connections 2024-04-26T20:32:27.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.214+0000 7f10fb600640 1 -- 172.21.15.102:0/2053762866 wait complete. 2024-04-26T20:32:27.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T20:32:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.452+0000 7fedbae00640 1 -- 172.21.15.102:0/885501746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedb41028a0 msgr2=0x7fedb4102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.452+0000 7fedbae00640 1 --2- 172.21.15.102:0/885501746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedb41028a0 0x7fedb4102d20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feda0009900 tx=0x7feda002a410 comp rx=0 tx=0).stop 2024-04-26T20:32:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.452+0000 7fedbae00640 1 -- 172.21.15.102:0/885501746 shutdown_connections 2024-04-26T20:32:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.452+0000 7fedbae00640 1 --2- 172.21.15.102:0/885501746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedb41028a0 0x7fedb4102d20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.452+0000 7fedbae00640 1 --2- 172.21.15.102:0/885501746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb4103260 0x7fedb4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.452+0000 7fedbae00640 1 --2- 172.21.15.102:0/885501746 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fedb41016b0 0x7fedb4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.452+0000 7fedbae00640 1 -- 172.21.15.102:0/885501746 >> 172.21.15.102:0/885501746 conn(0x7fedb40fcc00 msgr2=0x7fedb40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.453+0000 7fedbae00640 1 -- 172.21.15.102:0/885501746 shutdown_connections 2024-04-26T20:32:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.453+0000 7fedbae00640 1 -- 172.21.15.102:0/885501746 wait complete. 2024-04-26T20:32:27.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedbae00640 1 Processor -- start 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedbae00640 1 -- start start 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedbae00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fedb41016b0 0x7fedb4198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedbae00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb41028a0 0x7fedb4198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedbae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedb4103260 0x7fedb419ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedbae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fedb410d1e0 con 0x7fedb4103260 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedbae00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fedb410d060 con 0x7fedb41016b0 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedbae00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fedb410d360 con 0x7fedb41028a0 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedb9a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fedb41016b0 0x7fedb4198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedb9a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fedb41016b0 0x7fedb4198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:54970/0 (socket says 172.21.15.102:54970) 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedb9a00640 1 -- 172.21.15.102:0/197908083 learned_addr learned my addr 172.21.15.102:0/197908083 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.454+0000 7fedba400640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedb4103260 0x7fedb419ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedb9000640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb41028a0 0x7fedb4198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedb9a00640 1 -- 172.21.15.102:0/197908083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb41028a0 msgr2=0x7fedb4198f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedb9a00640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb41028a0 0x7fedb4198f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedb9a00640 1 -- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedb4103260 msgr2=0x7fedb419ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedb9a00640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedb4103260 0x7fedb419ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedb9a00640 1 -- 172.21.15.102:0/197908083 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb41a05d0 con 0x7fedb41016b0 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedba400640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedb4103260 0x7fedb419ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedb9a00640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fedb41016b0 0x7fedb4198a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feda800e9c0 tx=0x7feda800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedb2000640 1 -- 172.21.15.102:0/197908083 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda800cde0 con 0x7fedb41016b0 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedbae00640 1 -- 172.21.15.102:0/197908083 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb41a08c0 con 0x7fedb41016b0 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedb2000640 1 -- 172.21.15.102:0/197908083 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda80108e0 con 0x7fedb41016b0 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedb2000640 1 -- 172.21.15.102:0/197908083 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda80188d0 con 0x7fedb41016b0 2024-04-26T20:32:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.455+0000 7fedbae00640 1 -- 172.21.15.102:0/197908083 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb41a0e60 con 0x7fedb41016b0 2024-04-26T20:32:27.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.457+0000 7fedbae00640 1 -- 172.21.15.102:0/197908083 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedb41a1090 con 0x7fedb41016b0 2024-04-26T20:32:27.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.458+0000 7fedb2000640 1 -- 172.21.15.102:0/197908083 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7feda801f020 con 0x7fedb41016b0 2024-04-26T20:32:27.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.460+0000 7fedb2000640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fed9c0a3c30 0x7fed9c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.460+0000 7fedb2000640 1 -- 172.21.15.102:0/197908083 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7feda80d0650 con 0x7fedb41016b0 2024-04-26T20:32:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.460+0000 7fedb9000640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fed9c0a3c30 0x7fed9c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.461+0000 7fedb9000640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fed9c0a3c30 0x7fed9c0a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fedb4199ec0 tx=0x7feda00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:27.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.464+0000 7fedb2000640 1 -- 172.21.15.102:0/197908083 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7feda809cfa0 con 0x7fedb41016b0 2024-04-26T20:32:27.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:27 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T20:32:27.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:27 smithi130 ceph-mon[32704]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T20:32:27.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:27 smithi130 ceph-mon[32704]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T20:32:27.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:27 smithi130 ceph-mon[32704]: pgmap v313: 97 pgs: 12 creating+peering, 1 active+clean, 84 unknown; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:27.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:27 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2053762866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T20:32:27.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:27 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T20:32:27.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:27 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] up:active 2024-04-26T20:32:27.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:27 smithi130 ceph-mon[32704]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:27.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:27 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T20:32:27.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:27 smithi130 ceph-mon[32704]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T20:32:27.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:27 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T20:32:27.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:27 smithi102 ceph-mon[28094]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T20:32:27.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:27 smithi102 ceph-mon[28094]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T20:32:27.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:27 smithi102 ceph-mon[28094]: pgmap v313: 97 pgs: 12 creating+peering, 1 active+clean, 84 unknown; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:27.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:27 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2053762866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T20:32:27.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:27 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T20:32:27.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:27 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] up:active 2024-04-26T20:32:27.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:27 smithi102 ceph-mon[28094]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:27.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:27 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T20:32:27.650 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:27 smithi102 ceph-mon[28094]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T20:32:27.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:27.666+0000 7fedbae00640 1 -- 172.21.15.102:0/197908083 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fedb41a1260 con 0x7fedb41016b0 2024-04-26T20:32:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:27 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T20:32:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:27 smithi112 ceph-mon[32769]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T20:32:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:27 smithi112 ceph-mon[32769]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T20:32:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:27 smithi112 ceph-mon[32769]: pgmap v313: 97 pgs: 12 creating+peering, 1 active+clean, 84 unknown; 577 KiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:27 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2053762866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T20:32:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:27 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T20:32:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:27 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] up:active 2024-04-26T20:32:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:27 smithi112 ceph-mon[32769]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:27 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T20:32:27.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:27 smithi112 ceph-mon[32769]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T20:32:28.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:28 smithi130 ceph-mon[32704]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:32:28.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:28 smithi130 ceph-mon[32704]: Cluster is now healthy 2024-04-26T20:32:28.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:28 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/197908083' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:32:28.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:28 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:32:28.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:28 smithi102 ceph-mon[28094]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:32:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:28 smithi102 ceph-mon[28094]: Cluster is now healthy 2024-04-26T20:32:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:28 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/197908083' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:32:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:28 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:32:28.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:28 smithi112 ceph-mon[32769]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:32:28.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:28 smithi112 ceph-mon[32769]: Cluster is now healthy 2024-04-26T20:32:28.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:28 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/197908083' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:32:28.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:28 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T20:32:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.347+0000 7fedb2000640 1 -- 172.21.15.102:0/197908083 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7feda809c8a0 con 0x7fedb41016b0 2024-04-26T20:32:29.353 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data_ec' created 2024-04-26T20:32:29.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.355+0000 7fedbae00640 1 -- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fed9c0a3c30 msgr2=0x7fed9c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.355+0000 7fedbae00640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fed9c0a3c30 0x7fed9c0a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fedb4199ec0 tx=0x7feda00314a0 comp rx=0 tx=0).stop 2024-04-26T20:32:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.355+0000 7fedbae00640 1 -- 172.21.15.102:0/197908083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fedb41016b0 msgr2=0x7fedb4198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.355+0000 7fedbae00640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fedb41016b0 0x7fedb4198a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feda800e9c0 tx=0x7feda800ee80 comp rx=0 tx=0).stop 2024-04-26T20:32:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.355+0000 7fedbae00640 1 -- 172.21.15.102:0/197908083 shutdown_connections 2024-04-26T20:32:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.355+0000 7fedbae00640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fed9c0a3c30 0x7fed9c0a60f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.355+0000 7fedbae00640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedb4103260 0x7fedb419ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.355+0000 7fedbae00640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb41028a0 0x7fedb4198f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:29.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.355+0000 7fedbae00640 1 --2- 172.21.15.102:0/197908083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fedb41016b0 0x7fedb4198a00 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feda800e9c0 tx=0x7feda800ee80 comp rx=0 tx=0).stop 2024-04-26T20:32:29.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.355+0000 7fedbae00640 1 -- 172.21.15.102:0/197908083 >> 172.21.15.102:0/197908083 conn(0x7fedb40fcc00 msgr2=0x7fedb4069ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:29.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.357+0000 7fedbae00640 1 -- 172.21.15.102:0/197908083 shutdown_connections 2024-04-26T20:32:29.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.358+0000 7fedbae00640 1 -- 172.21.15.102:0/197908083 wait complete. 2024-04-26T20:32:29.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-26T20:32:29.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.562+0000 7fba77e00640 1 -- 172.21.15.102:0/2847965685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba78078ed0 msgr2=0x7fba78079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:29.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.562+0000 7fba77e00640 1 --2- 172.21.15.102:0/2847965685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba78078ed0 0x7fba78079770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fba70009ca0 tx=0x7fba7002a8c0 comp rx=0 tx=0).stop 2024-04-26T20:32:29.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.562+0000 7fba77e00640 1 -- 172.21.15.102:0/2847965685 shutdown_connections 2024-04-26T20:32:29.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.562+0000 7fba77e00640 1 --2- 172.21.15.102:0/2847965685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba78078530 0x7fba78078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:29.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.562+0000 7fba77e00640 1 --2- 172.21.15.102:0/2847965685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba78078ed0 0x7fba78079770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:29.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.562+0000 7fba77e00640 1 --2- 172.21.15.102:0/2847965685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba780772f0 0x7fba78077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:29.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.562+0000 7fba77e00640 1 -- 172.21.15.102:0/2847965685 >> 172.21.15.102:0/2847965685 conn(0x7fba780fffe0 msgr2=0x7fba78102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:29.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.562+0000 7fba77e00640 1 -- 172.21.15.102:0/2847965685 shutdown_connections 2024-04-26T20:32:29.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.562+0000 7fba77e00640 1 -- 172.21.15.102:0/2847965685 wait complete. 2024-04-26T20:32:29.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.563+0000 7fba77e00640 1 Processor -- start 2024-04-26T20:32:29.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.563+0000 7fba77e00640 1 -- start start 2024-04-26T20:32:29.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba77e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba780772f0 0x7fba7819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:29.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba77e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba78078530 0x7fba7819f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba77e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba78078ed0 0x7fba781a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba77e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fba781149b0 con 0x7fba78078530 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba77e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fba78114830 con 0x7fba780772f0 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba77e00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fba78114b30 con 0x7fba78078ed0 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba76000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba78078530 0x7fba7819f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba76000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba78078530 0x7fba7819f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44754/0 (socket says 172.21.15.102:44754) 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba76000640 1 -- 172.21.15.102:0/2131167548 learned_addr learned my addr 172.21.15.102:0/2131167548 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba77400640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba78078ed0 0x7fba781a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.564+0000 7fba76a00640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba780772f0 0x7fba7819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba76000640 1 -- 172.21.15.102:0/2131167548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba78078ed0 msgr2=0x7fba781a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:29.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba76000640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba78078ed0 0x7fba781a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba76000640 1 -- 172.21.15.102:0/2131167548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba780772f0 msgr2=0x7fba7819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba76000640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba780772f0 0x7fba7819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba76000640 1 -- 172.21.15.102:0/2131167548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba781a6c80 con 0x7fba78078530 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba77400640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba78078ed0 0x7fba781a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba76000640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba78078530 0x7fba7819f5f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fba6400e9f0 tx=0x7fba6400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba7ca00640 1 -- 172.21.15.102:0/2131167548 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6400cd80 con 0x7fba78078530 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba77e00640 1 -- 172.21.15.102:0/2131167548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba781a6f70 con 0x7fba78078530 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba7ca00640 1 -- 172.21.15.102:0/2131167548 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba640108e0 con 0x7fba78078530 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba7ca00640 1 -- 172.21.15.102:0/2131167548 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64018980 con 0x7fba78078530 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.565+0000 7fba77e00640 1 -- 172.21.15.102:0/2131167548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba781a7480 con 0x7fba78078530 2024-04-26T20:32:29.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.566+0000 7fba77e00640 1 -- 172.21.15.102:0/2131167548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba781113e0 con 0x7fba78078530 2024-04-26T20:32:29.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.569+0000 7fba7ca00640 1 -- 172.21.15.102:0/2131167548 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fba64010430 con 0x7fba78078530 2024-04-26T20:32:29.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.571+0000 7fba7ca00640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fba5c0a3bb0 0x7fba5c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:29.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.571+0000 7fba7ca00640 1 -- 172.21.15.102:0/2131167548 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fba640d0d70 con 0x7fba78078530 2024-04-26T20:32:29.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.571+0000 7fba76a00640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fba5c0a3bb0 0x7fba5c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:29.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.572+0000 7fba76a00640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fba5c0a3bb0 0x7fba5c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba68009bd0 tx=0x7fba68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:29.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.573+0000 7fba7ca00640 1 -- 172.21.15.102:0/2131167548 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fba6409ce80 con 0x7fba78078530 2024-04-26T20:32:29.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:29 smithi130 ceph-mon[32704]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T20:32:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:29 smithi130 ceph-mon[32704]: pgmap v316: 97 pgs: 12 creating+peering, 67 active+clean, 18 unknown; 578 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 2 op/s 2024-04-26T20:32:29.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:29 smithi102 ceph-mon[28094]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T20:32:29.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:29 smithi102 ceph-mon[28094]: pgmap v316: 97 pgs: 12 creating+peering, 67 active+clean, 18 unknown; 578 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 2 op/s 2024-04-26T20:32:29.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:29 smithi112 ceph-mon[32769]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T20:32:29.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:29 smithi112 ceph-mon[32769]: pgmap v316: 97 pgs: 12 creating+peering, 67 active+clean, 18 unknown; 578 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 2 op/s 2024-04-26T20:32:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:29.776+0000 7fba77e00640 1 -- 172.21.15.102:0/2131167548 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fba7802dad0 con 0x7fba78078530 2024-04-26T20:32:30.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.354+0000 7fba7ca00640 1 -- 172.21.15.102:0/2131167548 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fba640a1180 con 0x7fba78078530 2024-04-26T20:32:30.356 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 4 allow_ec_overwrites to true 2024-04-26T20:32:30.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.360+0000 7fba77e00640 1 -- 172.21.15.102:0/2131167548 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fba5c0a3bb0 msgr2=0x7fba5c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:30.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.360+0000 7fba77e00640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fba5c0a3bb0 0x7fba5c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba68009bd0 tx=0x7fba68005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:30.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.360+0000 7fba77e00640 1 -- 172.21.15.102:0/2131167548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba78078530 msgr2=0x7fba7819f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:30.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.360+0000 7fba77e00640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba78078530 0x7fba7819f5f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fba6400e9f0 tx=0x7fba6400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:32:30.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.361+0000 7fba77e00640 1 -- 172.21.15.102:0/2131167548 shutdown_connections 2024-04-26T20:32:30.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.361+0000 7fba77e00640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fba5c0a3bb0 0x7fba5c0a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:30.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.361+0000 7fba77e00640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba78078530 0x7fba7819f5f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:30.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.361+0000 7fba77e00640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba78078ed0 0x7fba781a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:30.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.361+0000 7fba77e00640 1 --2- 172.21.15.102:0/2131167548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba780772f0 0x7fba7819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:30.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.361+0000 7fba77e00640 1 -- 172.21.15.102:0/2131167548 >> 172.21.15.102:0/2131167548 conn(0x7fba780fffe0 msgr2=0x7fba78103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:30.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.362+0000 7fba77e00640 1 -- 172.21.15.102:0/2131167548 shutdown_connections 2024-04-26T20:32:30.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.363+0000 7fba77e00640 1 -- 172.21.15.102:0/2131167548 wait complete. 2024-04-26T20:32:30.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-26T20:32:30.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.574+0000 7f9666200640 1 -- 172.21.15.102:0/3519403822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9660104b90 msgr2=0x7f9660104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:30.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.574+0000 7f9666200640 1 --2- 172.21.15.102:0/3519403822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9660104b90 0x7f9660104fb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9650009860 tx=0x7f965002a340 comp rx=0 tx=0).stop 2024-04-26T20:32:30.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.574+0000 7f9666200640 1 -- 172.21.15.102:0/3519403822 shutdown_connections 2024-04-26T20:32:30.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.574+0000 7f9666200640 1 --2- 172.21.15.102:0/3519403822 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9660106740 0x7f966010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:30.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.574+0000 7f9666200640 1 --2- 172.21.15.102:0/3519403822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9660105d80 0x7f9660106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:30.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.574+0000 7f9666200640 1 --2- 172.21.15.102:0/3519403822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9660104b90 0x7f9660104fb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:30.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.574+0000 7f9666200640 1 -- 172.21.15.102:0/3519403822 >> 172.21.15.102:0/3519403822 conn(0x7f9660100120 msgr2=0x7f9660102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:30.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.574+0000 7f9666200640 1 -- 172.21.15.102:0/3519403822 shutdown_connections 2024-04-26T20:32:30.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.574+0000 7f9666200640 1 -- 172.21.15.102:0/3519403822 wait complete. 2024-04-26T20:32:30.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f9666200640 1 Processor -- start 2024-04-26T20:32:30.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f9666200640 1 -- start start 2024-04-26T20:32:30.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f9666200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9660104b90 0x7f966019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:30.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f9666200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9660105d80 0x7f966019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:30.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f9666200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9660106740 0x7f96601a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:30.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f9666200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9660110690 con 0x7f9660105d80 2024-04-26T20:32:30.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f9666200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9660110510 con 0x7f9660104b90 2024-04-26T20:32:30.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f9666200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9660110810 con 0x7f9660106740 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f965ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9660105d80 0x7f966019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f965ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9660105d80 0x7f966019b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44784/0 (socket says 172.21.15.102:44784) 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f965ea00640 1 -- 172.21.15.102:0/693872244 learned_addr learned my addr 172.21.15.102:0/693872244 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.575+0000 7f965f400640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9660104b90 0x7f966019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f965fe00640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9660106740 0x7f96601a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f965ea00640 1 -- 172.21.15.102:0/693872244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9660106740 msgr2=0x7f96601a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f965ea00640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9660106740 0x7f96601a23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f965ea00640 1 -- 172.21.15.102:0/693872244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9660104b90 msgr2=0x7f966019ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f965ea00640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9660104b90 0x7f966019ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f965ea00640 1 -- 172.21.15.102:0/693872244 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96601a2a40 con 0x7f9660105d80 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f965fe00640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9660106740 0x7f96601a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f965ea00640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9660105d80 0x7f966019b360 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f964c00e9f0 tx=0x7f964c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f9657e00640 1 -- 172.21.15.102:0/693872244 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c00cd80 con 0x7f9660105d80 2024-04-26T20:32:30.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f9657e00640 1 -- 172.21.15.102:0/693872244 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f964c0108e0 con 0x7f9660105d80 2024-04-26T20:32:30.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f9657e00640 1 -- 172.21.15.102:0/693872244 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c018980 con 0x7f9660105d80 2024-04-26T20:32:30.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.576+0000 7f9666200640 1 -- 172.21.15.102:0/693872244 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96601a2d30 con 0x7f9660105d80 2024-04-26T20:32:30.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.577+0000 7f9666200640 1 -- 172.21.15.102:0/693872244 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96601a3270 con 0x7f9660105d80 2024-04-26T20:32:30.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.578+0000 7f9666200640 1 -- 172.21.15.102:0/693872244 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f966010bfc0 con 0x7f9660105d80 2024-04-26T20:32:30.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.582+0000 7f9657e00640 1 -- 172.21.15.102:0/693872244 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f964c010430 con 0x7f9660105d80 2024-04-26T20:32:30.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.584+0000 7f9657e00640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96440a3c30 0x7f96440a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:30.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.584+0000 7f9657e00640 1 -- 172.21.15.102:0/693872244 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f964c0d0d80 con 0x7f9660105d80 2024-04-26T20:32:30.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.584+0000 7f965f400640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96440a3c30 0x7f96440a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:30.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.585+0000 7f965f400640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96440a3c30 0x7f96440a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f965002a930 tx=0x7f9650030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:30.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.586+0000 7f9657e00640 1 -- 172.21.15.102:0/693872244 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f964c09ce90 con 0x7f9660105d80 2024-04-26T20:32:30.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:30 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T20:32:30.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:30 smithi130 ceph-mon[32704]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T20:32:30.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:30 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2131167548' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T20:32:30.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:30 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T20:32:30.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:30 smithi102 ceph-mon[28094]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T20:32:30.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:30 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2131167548' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T20:32:30.676 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:30 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T20:32:30.676 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:30 smithi112 ceph-mon[32769]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T20:32:30.676 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:30 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2131167548' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T20:32:30.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:30.825+0000 7f9666200640 1 -- 172.21.15.102:0/693872244 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f966002dad0 con 0x7f9660105d80 2024-04-26T20:32:31.026 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:30 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b[62857]: 2024-04-26T20:32:30.674+0000 7f74d1749700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T20:32:31.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.370+0000 7f9657e00640 1 -- 172.21.15.102:0/693872244 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7f964c0a1190 con 0x7f9660105d80 2024-04-26T20:32:31.372 INFO:teuthology.orchestra.run.smithi102.stderr:added data pool 4 to fsmap 2024-04-26T20:32:31.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.374+0000 7f9666200640 1 -- 172.21.15.102:0/693872244 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96440a3c30 msgr2=0x7f96440a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:31.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.374+0000 7f9666200640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96440a3c30 0x7f96440a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f965002a930 tx=0x7f9650030470 comp rx=0 tx=0).stop 2024-04-26T20:32:31.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.374+0000 7f9666200640 1 -- 172.21.15.102:0/693872244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9660105d80 msgr2=0x7f966019b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.374+0000 7f9666200640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9660105d80 0x7f966019b360 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f964c00e9f0 tx=0x7f964c00eeb0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.374+0000 7f9666200640 1 -- 172.21.15.102:0/693872244 shutdown_connections 2024-04-26T20:32:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.374+0000 7f9666200640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f96440a3c30 0x7f96440a60f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.374+0000 7f9666200640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9660105d80 0x7f966019b360 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.374+0000 7f9666200640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9660106740 0x7f96601a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.374+0000 7f9666200640 1 --2- 172.21.15.102:0/693872244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9660104b90 0x7f966019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.374+0000 7f9666200640 1 -- 172.21.15.102:0/693872244 >> 172.21.15.102:0/693872244 conn(0x7f9660100120 msgr2=0x7f9660109190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.375+0000 7f9666200640 1 -- 172.21.15.102:0/693872244 shutdown_connections 2024-04-26T20:32:31.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.375+0000 7f9666200640 1 -- 172.21.15.102:0/693872244 wait complete. 2024-04-26T20:32:31.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.602+0000 7f06bdc00640 1 -- 172.21.15.102:0/3361256457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b8104b90 msgr2=0x7f06b8104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.602+0000 7f06bdc00640 1 --2- 172.21.15.102:0/3361256457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b8104b90 0x7f06b8104fb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009900 tx=0x7f06a802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.603+0000 7f06bdc00640 1 -- 172.21.15.102:0/3361256457 shutdown_connections 2024-04-26T20:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.603+0000 7f06bdc00640 1 --2- 172.21.15.102:0/3361256457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06b8106740 0x7f06b810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.603+0000 7f06bdc00640 1 --2- 172.21.15.102:0/3361256457 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06b8105d80 0x7f06b8106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.603+0000 7f06bdc00640 1 --2- 172.21.15.102:0/3361256457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b8104b90 0x7f06b8104fb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.603+0000 7f06bdc00640 1 -- 172.21.15.102:0/3361256457 >> 172.21.15.102:0/3361256457 conn(0x7f06b8100120 msgr2=0x7f06b8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.603+0000 7f06bdc00640 1 -- 172.21.15.102:0/3361256457 shutdown_connections 2024-04-26T20:32:31.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.603+0000 7f06bdc00640 1 -- 172.21.15.102:0/3361256457 wait complete. 2024-04-26T20:32:31.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.604+0000 7f06bdc00640 1 Processor -- start 2024-04-26T20:32:31.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.604+0000 7f06bdc00640 1 -- start start 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.604+0000 7f06bdc00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b8104b90 0x7f06b819adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.604+0000 7f06bdc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06b8105d80 0x7f06b819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.604+0000 7f06bdc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06b8106740 0x7f06b81a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.604+0000 7f06bdc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f06b8110700 con 0x7f06b8106740 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.604+0000 7f06bdc00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f06b8110580 con 0x7f06b8105d80 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.604+0000 7f06bdc00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f06b8110880 con 0x7f06b8104b90 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06b7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06b8106740 0x7f06b81a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06ae000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06b8105d80 0x7f06b819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06b7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06b8106740 0x7f06b81a2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44808/0 (socket says 172.21.15.102:44808) 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06b7400640 1 -- 172.21.15.102:0/320403616 learned_addr learned my addr 172.21.15.102:0/320403616 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06b6a00640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b8104b90 0x7f06b819adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06b7400640 1 -- 172.21.15.102:0/320403616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b8104b90 msgr2=0x7f06b819adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06b7400640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b8104b90 0x7f06b819adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06b7400640 1 -- 172.21.15.102:0/320403616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06b8105d80 msgr2=0x7f06b819b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06b7400640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06b8105d80 0x7f06b819b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06b7400640 1 -- 172.21.15.102:0/320403616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b81a29d0 con 0x7f06b8106740 2024-04-26T20:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.605+0000 7f06b7400640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06b8106740 0x7f06b81a2330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f06b000eb10 tx=0x7f06b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.606+0000 7f06afe00640 1 -- 172.21.15.102:0/320403616 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0004b20 con 0x7f06b8106740 2024-04-26T20:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.606+0000 7f06bdc00640 1 -- 172.21.15.102:0/320403616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b81a2cc0 con 0x7f06b8106740 2024-04-26T20:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.606+0000 7f06bdc00640 1 -- 172.21.15.102:0/320403616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b81a31c0 con 0x7f06b8106740 2024-04-26T20:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.606+0000 7f06afe00640 1 -- 172.21.15.102:0/320403616 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06b0004cc0 con 0x7f06b8106740 2024-04-26T20:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.606+0000 7f06afe00640 1 -- 172.21.15.102:0/320403616 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0010690 con 0x7f06b8106740 2024-04-26T20:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.607+0000 7f06bdc00640 1 -- 172.21.15.102:0/320403616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b810beb0 con 0x7f06b8106740 2024-04-26T20:32:31.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.609+0000 7f06afe00640 1 -- 172.21.15.102:0/320403616 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f06b0005000 con 0x7f06b8106740 2024-04-26T20:32:31.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.610+0000 7f06afe00640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f069c0a3c30 0x7f069c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:31.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.610+0000 7f06afe00640 1 -- 172.21.15.102:0/320403616 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f06b00d0c80 con 0x7f06b8106740 2024-04-26T20:32:31.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.610+0000 7f06b6a00640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f069c0a3c30 0x7f069c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:31.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.611+0000 7f06b6a00640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f069c0a3c30 0x7f069c0a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f06b819c290 tx=0x7f06a80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:31.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.614+0000 7f06afe00640 1 -- 172.21.15.102:0/320403616 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f06b009d420 con 0x7f06b8106740 2024-04-26T20:32:31.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:31 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2131167548' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T20:32:31.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:31 smithi102 ceph-mon[28094]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T20:32:31.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:31 smithi102 ceph-mon[28094]: pgmap v319: 129 pgs: 12 creating+peering, 97 active+clean, 20 unknown; 579 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-04-26T20:32:31.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:31 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/693872244' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T20:32:31.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:31 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2131167548' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T20:32:31.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:31 smithi112 ceph-mon[32769]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T20:32:31.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:31 smithi112 ceph-mon[32769]: pgmap v319: 129 pgs: 12 creating+peering, 97 active+clean, 20 unknown; 579 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-04-26T20:32:31.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:31 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/693872244' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T20:32:31.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.859+0000 7f06bdc00640 1 -- 172.21.15.102:0/320403616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06b802dad0 con 0x7f06b8106740 2024-04-26T20:32:31.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.860+0000 7f06afe00640 1 -- 172.21.15.102:0/320403616 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11818 (secure 0 0 0) 0x7f06b009cd20 con 0x7f06b8106740 2024-04-26T20:32:31.861 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:31.861 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T20:32:25.769708+0000","modified":"2024-04-26T20:32:30.827828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24593,"qdb_cluster":[24593]},"id":1}]} 2024-04-26T20:32:31.862 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 16 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.862+0000 7f06bdc00640 1 -- 172.21.15.102:0/320403616 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f069c0a3c30 msgr2=0x7f069c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.862+0000 7f06bdc00640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f069c0a3c30 0x7f069c0a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f06b819c290 tx=0x7f06a80314a0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.862+0000 7f06bdc00640 1 -- 172.21.15.102:0/320403616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06b8106740 msgr2=0x7f06b81a2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.862+0000 7f06bdc00640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06b8106740 0x7f06b81a2330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f06b000eb10 tx=0x7f06b000c2d0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.863+0000 7f06bdc00640 1 -- 172.21.15.102:0/320403616 shutdown_connections 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.863+0000 7f06bdc00640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f069c0a3c30 0x7f069c0a60f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.863+0000 7f06bdc00640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06b8106740 0x7f06b81a2330 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.863+0000 7f06bdc00640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06b8105d80 0x7f06b819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.863+0000 7f06bdc00640 1 --2- 172.21.15.102:0/320403616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06b8104b90 0x7f06b819adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.863+0000 7f06bdc00640 1 -- 172.21.15.102:0/320403616 >> 172.21.15.102:0/320403616 conn(0x7f06b8100120 msgr2=0x7f06b8109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.863+0000 7f06bdc00640 1 -- 172.21.15.102:0/320403616 shutdown_connections 2024-04-26T20:32:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:31.863+0000 7f06bdc00640 1 -- 172.21.15.102:0/320403616 wait complete. 2024-04-26T20:32:31.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:31 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2131167548' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T20:32:31.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:31 smithi130 ceph-mon[32704]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T20:32:31.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:31 smithi130 ceph-mon[32704]: pgmap v319: 129 pgs: 12 creating+peering, 97 active+clean, 20 unknown; 579 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-04-26T20:32:31.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:31 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/693872244' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T20:32:31.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:32:32.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.074+0000 7f444e800640 1 -- 172.21.15.102:0/4053843798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4448102970 msgr2=0x7f4448102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.074+0000 7f444e800640 1 --2- 172.21.15.102:0/4053843798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4448102970 0x7f4448102d90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4438009ca0 tx=0x7f443802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.075+0000 7f444e800640 1 -- 172.21.15.102:0/4053843798 shutdown_connections 2024-04-26T20:32:32.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.075+0000 7f444e800640 1 --2- 172.21.15.102:0/4053843798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4448104520 0x7f444810adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.075+0000 7f444e800640 1 --2- 172.21.15.102:0/4053843798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4448103b60 0x7f4448103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.075+0000 7f444e800640 1 --2- 172.21.15.102:0/4053843798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4448102970 0x7f4448102d90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.075+0000 7f444e800640 1 -- 172.21.15.102:0/4053843798 >> 172.21.15.102:0/4053843798 conn(0x7f44480fdee0 msgr2=0x7f4448100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:32.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.075+0000 7f444e800640 1 -- 172.21.15.102:0/4053843798 shutdown_connections 2024-04-26T20:32:32.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.075+0000 7f444e800640 1 -- 172.21.15.102:0/4053843798 wait complete. 2024-04-26T20:32:32.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f444e800640 1 Processor -- start 2024-04-26T20:32:32.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f444e800640 1 -- start start 2024-04-26T20:32:32.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f444e800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4448102970 0x7f444819ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f444e800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4448103b60 0x7f444819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f444e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4448104520 0x7f44481a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f444e800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f444810e4e0 con 0x7f4448104520 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f444e800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f444810e360 con 0x7f4448103b60 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f444e800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f444810e660 con 0x7f4448102970 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f4447e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4448104520 0x7f44481a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f4446a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4448103b60 0x7f444819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f4447400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4448102970 0x7f444819ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f4446a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4448103b60 0x7f444819b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:55036/0 (socket says 172.21.15.102:55036) 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.076+0000 7f4446a00640 1 -- 172.21.15.102:0/645267638 learned_addr learned my addr 172.21.15.102:0/645267638 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f4446a00640 1 -- 172.21.15.102:0/645267638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4448102970 msgr2=0x7f444819ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f4446a00640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4448102970 0x7f444819ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f4446a00640 1 -- 172.21.15.102:0/645267638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4448104520 msgr2=0x7f44481a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f4446a00640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4448104520 0x7f44481a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f4446a00640 1 -- 172.21.15.102:0/645267638 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44481a29e0 con 0x7f4448103b60 2024-04-26T20:32:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f4446a00640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4448103b60 0x7f444819b3a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f443000e9f0 tx=0x7f443000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f443fe00640 1 -- 172.21.15.102:0/645267638 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443000cd80 con 0x7f4448103b60 2024-04-26T20:32:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f444e800640 1 -- 172.21.15.102:0/645267638 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44481a2cd0 con 0x7f4448103b60 2024-04-26T20:32:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f443fe00640 1 -- 172.21.15.102:0/645267638 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44300054d0 con 0x7f4448103b60 2024-04-26T20:32:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f443fe00640 1 -- 172.21.15.102:0/645267638 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4430010690 con 0x7f4448103b60 2024-04-26T20:32:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.077+0000 7f444e800640 1 -- 172.21.15.102:0/645267638 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44481a31d0 con 0x7f4448103b60 2024-04-26T20:32:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.078+0000 7f444e800640 1 -- 172.21.15.102:0/645267638 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44481a3460 con 0x7f4448103b60 2024-04-26T20:32:32.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.081+0000 7f443fe00640 1 -- 172.21.15.102:0/645267638 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f4430004e30 con 0x7f4448103b60 2024-04-26T20:32:32.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.082+0000 7f443fe00640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f442c0a3bb0 0x7f442c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.082+0000 7f443fe00640 1 -- 172.21.15.102:0/645267638 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f44300d0c80 con 0x7f4448103b60 2024-04-26T20:32:32.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.082+0000 7f4447400640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f442c0a3bb0 0x7f442c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.083+0000 7f4447400640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f442c0a3bb0 0x7f442c0a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4438000f80 tx=0x7f4438005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:32.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.084+0000 7f443fe00640 1 -- 172.21.15.102:0/645267638 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4430054060 con 0x7f4448103b60 2024-04-26T20:32:32.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.288+0000 7f444e800640 1 -- 172.21.15.102:0/645267638 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44481a3630 con 0x7f4448103b60 2024-04-26T20:32:32.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.290+0000 7f443fe00640 1 -- 172.21.15.102:0/645267638 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24164 (secure 0 0 0) 0x7f443009d3c0 con 0x7f4448103b60 2024-04-26T20:32:32.292 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:32.292 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:32:30.827511+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:32:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 -- 172.21.15.102:0/645267638 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f442c0a3bb0 msgr2=0x7f442c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f442c0a3bb0 0x7f442c0a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4438000f80 tx=0x7f4438005f30 comp rx=0 tx=0).stop 2024-04-26T20:32:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 -- 172.21.15.102:0/645267638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4448103b60 msgr2=0x7f444819b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4448103b60 0x7f444819b3a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f443000e9f0 tx=0x7f443000edf0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 -- 172.21.15.102:0/645267638 shutdown_connections 2024-04-26T20:32:32.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f442c0a3bb0 0x7f442c0a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4448104520 0x7f44481a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4448103b60 0x7f444819b3a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 --2- 172.21.15.102:0/645267638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4448102970 0x7f444819ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 -- 172.21.15.102:0/645267638 >> 172.21.15.102:0/645267638 conn(0x7f44480fdee0 msgr2=0x7f4448074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 -- 172.21.15.102:0/645267638 shutdown_connections 2024-04-26T20:32:32.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.293+0000 7f444e800640 1 -- 172.21.15.102:0/645267638 wait complete. 2024-04-26T20:32:32.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:32:32.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.508+0000 7f25f0a00640 1 -- 172.21.15.102:0/2851666661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ec102750 msgr2=0x7f25ec102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.508+0000 7f25f0a00640 1 --2- 172.21.15.102:0/2851666661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ec102750 0x7f25ec102b70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f25dc009900 tx=0x7f25dc02a410 comp rx=0 tx=0).stop 2024-04-26T20:32:32.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.509+0000 7f25f0a00640 1 -- 172.21.15.102:0/2851666661 shutdown_connections 2024-04-26T20:32:32.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.509+0000 7f25f0a00640 1 --2- 172.21.15.102:0/2851666661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25ec104300 0x7f25ec10ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.509+0000 7f25f0a00640 1 --2- 172.21.15.102:0/2851666661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25ec103940 0x7f25ec103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.509+0000 7f25f0a00640 1 --2- 172.21.15.102:0/2851666661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ec102750 0x7f25ec102b70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.509+0000 7f25f0a00640 1 -- 172.21.15.102:0/2851666661 >> 172.21.15.102:0/2851666661 conn(0x7f25ec0fdcc0 msgr2=0x7f25ec100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:32.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.509+0000 7f25f0a00640 1 -- 172.21.15.102:0/2851666661 shutdown_connections 2024-04-26T20:32:32.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.509+0000 7f25f0a00640 1 -- 172.21.15.102:0/2851666661 wait complete. 2024-04-26T20:32:32.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.510+0000 7f25f0a00640 1 Processor -- start 2024-04-26T20:32:32.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.510+0000 7f25f0a00640 1 -- start start 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.510+0000 7f25f0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ec102750 0x7f25ec19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.510+0000 7f25f0a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25ec103940 0x7f25ec19b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.510+0000 7f25f0a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25ec104300 0x7f25ec1a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.510+0000 7f25f0a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec10e2d0 con 0x7f25ec102750 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.510+0000 7f25f0a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec10e150 con 0x7f25ec104300 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.510+0000 7f25f0a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f25ec10e450 con 0x7f25ec103940 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.510+0000 7f25eb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ec102750 0x7f25ec19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.510+0000 7f25eb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ec102750 0x7f25ec19ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44842/0 (socket says 172.21.15.102:44842) 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.511+0000 7f25eb400640 1 -- 172.21.15.102:0/3530694318 learned_addr learned my addr 172.21.15.102:0/3530694318 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.511+0000 7f25ebe00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25ec104300 0x7f25ec1a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.511+0000 7f25eaa00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25ec103940 0x7f25ec19b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.511+0000 7f25eb400640 1 -- 172.21.15.102:0/3530694318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25ec103940 msgr2=0x7f25ec19b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.511+0000 7f25eb400640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25ec103940 0x7f25ec19b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.511+0000 7f25eb400640 1 -- 172.21.15.102:0/3530694318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25ec104300 msgr2=0x7f25ec1a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.511+0000 7f25eb400640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25ec104300 0x7f25ec1a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.511+0000 7f25eb400640 1 -- 172.21.15.102:0/3530694318 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25ec1a27b0 con 0x7f25ec102750 2024-04-26T20:32:32.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.511+0000 7f25eb400640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ec102750 0x7f25ec19ab90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f25dc031500 tx=0x7f25dc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:32.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.511+0000 7f25e3e00640 1 -- 172.21.15.102:0/3530694318 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc03f070 con 0x7f25ec102750 2024-04-26T20:32:32.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.512+0000 7f25e3e00640 1 -- 172.21.15.102:0/3530694318 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25dc031c50 con 0x7f25ec102750 2024-04-26T20:32:32.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.512+0000 7f25e3e00640 1 -- 172.21.15.102:0/3530694318 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc035030 con 0x7f25ec102750 2024-04-26T20:32:32.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.512+0000 7f25f0a00640 1 -- 172.21.15.102:0/3530694318 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25ec1a2a40 con 0x7f25ec102750 2024-04-26T20:32:32.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.512+0000 7f25f0a00640 1 -- 172.21.15.102:0/3530694318 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25ec1a2f80 con 0x7f25ec102750 2024-04-26T20:32:32.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.513+0000 7f25f0a00640 1 -- 172.21.15.102:0/3530694318 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25ec1a31b0 con 0x7f25ec102750 2024-04-26T20:32:32.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.515+0000 7f25e3e00640 1 -- 172.21.15.102:0/3530694318 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f25dc042070 con 0x7f25ec102750 2024-04-26T20:32:32.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.517+0000 7f25e3e00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f25d00a3c30 0x7f25d00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.517+0000 7f25e3e00640 1 -- 172.21.15.102:0/3530694318 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f25dc0f0d80 con 0x7f25ec102750 2024-04-26T20:32:32.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.517+0000 7f25eaa00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f25d00a3c30 0x7f25d00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.518+0000 7f25eaa00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f25d00a3c30 0x7f25d00a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f25ec19c0a0 tx=0x7f25d800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.522+0000 7f25e3e00640 1 -- 172.21.15.102:0/3530694318 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f25dc0bce70 con 0x7f25ec102750 2024-04-26T20:32:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:32 smithi102 ceph-mon[28094]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:32:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:32 smithi102 ceph-mon[28094]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T20:32:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:32 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/693872244' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T20:32:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:32 smithi102 ceph-mon[28094]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:32 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/320403616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:32.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:32 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/645267638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.714+0000 7f25f0a00640 1 -- 172.21.15.102:0/3530694318 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25ec1a34a0 con 0x7f25ec102750 2024-04-26T20:32:32.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.719+0000 7f25e3e00640 1 -- 172.21.15.102:0/3530694318 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24164 (secure 0 0 0) 0x7f25dc0c1170 con 0x7f25ec102750 2024-04-26T20:32:32.720 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:32.720 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:32:30.827511+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:32:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 -- 172.21.15.102:0/3530694318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f25d00a3c30 msgr2=0x7f25d00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f25d00a3c30 0x7f25d00a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f25ec19c0a0 tx=0x7f25d800b410 comp rx=0 tx=0).stop 2024-04-26T20:32:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 -- 172.21.15.102:0/3530694318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ec102750 msgr2=0x7f25ec19ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ec102750 0x7f25ec19ab90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f25dc031500 tx=0x7f25dc003c10 comp rx=0 tx=0).stop 2024-04-26T20:32:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 -- 172.21.15.102:0/3530694318 shutdown_connections 2024-04-26T20:32:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f25d00a3c30 0x7f25d00a60f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25ec104300 0x7f25ec1a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25ec103940 0x7f25ec19b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 --2- 172.21.15.102:0/3530694318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ec102750 0x7f25ec19ab90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 -- 172.21.15.102:0/3530694318 >> 172.21.15.102:0/3530694318 conn(0x7f25ec0fdcc0 msgr2=0x7f25ec074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 -- 172.21.15.102:0/3530694318 shutdown_connections 2024-04-26T20:32:32.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.721+0000 7f25f0a00640 1 -- 172.21.15.102:0/3530694318 wait complete. 2024-04-26T20:32:32.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:32.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:32 smithi112 ceph-mon[32769]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:32:32.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:32 smithi112 ceph-mon[32769]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T20:32:32.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:32 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/693872244' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T20:32:32.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:32 smithi112 ceph-mon[32769]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:32.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:32 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/320403616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:32.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:32 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/645267638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:32.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:32 smithi130 ceph-mon[32704]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T20:32:32.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:32 smithi130 ceph-mon[32704]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T20:32:32.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:32 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/693872244' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T20:32:32.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:32 smithi130 ceph-mon[32704]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:32.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:32 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/320403616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:32.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:32 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/645267638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:32.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.959+0000 7f9faa800640 1 -- 172.21.15.102:0/3612481720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9fa4075330 msgr2=0x7f9fa4075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.959+0000 7f9faa800640 1 --2- 172.21.15.102:0/3612481720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9fa4075330 0x7f9fa4075790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009900 tx=0x7f9f9402a480 comp rx=0 tx=0).stop 2024-04-26T20:32:32.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.959+0000 7f9faa800640 1 -- 172.21.15.102:0/3612481720 shutdown_connections 2024-04-26T20:32:32.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.959+0000 7f9faa800640 1 --2- 172.21.15.102:0/3612481720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fa410a600 0x7f9fa410caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.959+0000 7f9faa800640 1 --2- 172.21.15.102:0/3612481720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9fa4075330 0x7f9fa4075790 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.959+0000 7f9faa800640 1 --2- 172.21.15.102:0/3612481720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4076910 0x7f9fa4074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.959+0000 7f9faa800640 1 -- 172.21.15.102:0/3612481720 >> 172.21.15.102:0/3612481720 conn(0x7f9fa40fdcc0 msgr2=0x7f9fa4100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:32.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.959+0000 7f9faa800640 1 -- 172.21.15.102:0/3612481720 shutdown_connections 2024-04-26T20:32:32.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.959+0000 7f9faa800640 1 -- 172.21.15.102:0/3612481720 wait complete. 2024-04-26T20:32:32.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.960+0000 7f9faa800640 1 Processor -- start 2024-04-26T20:32:32.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.960+0000 7f9faa800640 1 -- start start 2024-04-26T20:32:32.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9faa800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9fa4075330 0x7f9fa419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9faa800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fa4076910 0x7f9fa419b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9faa800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa410a600 0x7f9fa41a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9faa800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa41104a0 con 0x7f9fa4076910 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9faa800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa4110320 con 0x7f9fa410a600 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9faa800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa4110620 con 0x7f9fa4075330 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9fa8a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fa4076910 0x7f9fa419b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9fa8a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fa4076910 0x7f9fa419b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44860/0 (socket says 172.21.15.102:44860) 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9fa8a00640 1 -- 172.21.15.102:0/1974450649 learned_addr learned my addr 172.21.15.102:0/1974450649 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9fa9e00640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa410a600 0x7f9fa41a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9fa9400640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9fa4075330 0x7f9fa419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9fa8a00640 1 -- 172.21.15.102:0/1974450649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9fa4075330 msgr2=0x7f9fa419ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9fa8a00640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9fa4075330 0x7f9fa419ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9fa8a00640 1 -- 172.21.15.102:0/1974450649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa410a600 msgr2=0x7f9fa41a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.961+0000 7f9fa8a00640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa410a600 0x7f9fa41a20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:32.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.962+0000 7f9fa8a00640 1 -- 172.21.15.102:0/1974450649 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa41a2740 con 0x7f9fa4076910 2024-04-26T20:32:32.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.962+0000 7f9fa9e00640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa410a600 0x7f9fa41a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:32.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.962+0000 7f9fa8a00640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fa4076910 0x7f9fa419b060 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9f94005940 tx=0x7f9f940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:32.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.962+0000 7f9fa2000640 1 -- 172.21.15.102:0/1974450649 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9403f030 con 0x7f9fa4076910 2024-04-26T20:32:32.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.962+0000 7f9fa2000640 1 -- 172.21.15.102:0/1974450649 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f94003cb0 con 0x7f9fa4076910 2024-04-26T20:32:32.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.962+0000 7f9faa800640 1 -- 172.21.15.102:0/1974450649 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa41a29d0 con 0x7f9fa4076910 2024-04-26T20:32:32.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.962+0000 7f9fa2000640 1 -- 172.21.15.102:0/1974450649 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94035070 con 0x7f9fa4076910 2024-04-26T20:32:32.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.962+0000 7f9faa800640 1 -- 172.21.15.102:0/1974450649 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa41a2ec0 con 0x7f9fa4076910 2024-04-26T20:32:32.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.963+0000 7f9faa800640 1 -- 172.21.15.102:0/1974450649 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa41093c0 con 0x7f9fa4076910 2024-04-26T20:32:32.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.965+0000 7f9fa2000640 1 -- 172.21.15.102:0/1974450649 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9f94042030 con 0x7f9fa4076910 2024-04-26T20:32:32.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.966+0000 7f9fa2000640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9f8c0a3bb0 0x7f9f8c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:32.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.967+0000 7f9fa2000640 1 -- 172.21.15.102:0/1974450649 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9f940f0a50 con 0x7f9fa4076910 2024-04-26T20:32:32.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.967+0000 7f9fa9400640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9f8c0a3bb0 0x7f9f8c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:32.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.967+0000 7f9fa9400640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9f8c0a3bb0 0x7f9f8c0a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c005fd0 tx=0x7f9f9c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:32.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:32.971+0000 7f9fa2000640 1 -- 172.21.15.102:0/1974450649 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9f94003e60 con 0x7f9fa4076910 2024-04-26T20:32:33.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.219+0000 7f9faa800640 1 -- 172.21.15.102:0/1974450649 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fa4063560 con 0x7f9fa4076910 2024-04-26T20:32:33.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.220+0000 7f9fa2000640 1 -- 172.21.15.102:0/1974450649 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11818 (secure 0 0 0) 0x7f9f940bcb40 con 0x7f9fa4076910 2024-04-26T20:32:33.222 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:33.222 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24526,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T20:32:25.769708+0000","modified":"2024-04-26T20:32:30.827828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24593,"qdb_cluster":[24593]},"id":1}]} 2024-04-26T20:32:33.222 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 16 2024-04-26T20:32:33.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.223+0000 7f9faa800640 1 -- 172.21.15.102:0/1974450649 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9f8c0a3bb0 msgr2=0x7f9f8c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.223+0000 7f9faa800640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9f8c0a3bb0 0x7f9f8c0a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c005fd0 tx=0x7f9f9c009500 comp rx=0 tx=0).stop 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.223+0000 7f9faa800640 1 -- 172.21.15.102:0/1974450649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fa4076910 msgr2=0x7f9fa419b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.223+0000 7f9faa800640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fa4076910 0x7f9fa419b060 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9f94005940 tx=0x7f9f940042f0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.224+0000 7f9faa800640 1 -- 172.21.15.102:0/1974450649 shutdown_connections 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.224+0000 7f9faa800640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9f8c0a3bb0 0x7f9f8c0a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.224+0000 7f9faa800640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fa4076910 0x7f9fa419b060 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.224+0000 7f9faa800640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa410a600 0x7f9fa41a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.224+0000 7f9faa800640 1 --2- 172.21.15.102:0/1974450649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9fa4075330 0x7f9fa419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.224+0000 7f9faa800640 1 -- 172.21.15.102:0/1974450649 >> 172.21.15.102:0/1974450649 conn(0x7f9fa40fdcc0 msgr2=0x7f9fa4106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.224+0000 7f9faa800640 1 -- 172.21.15.102:0/1974450649 shutdown_connections 2024-04-26T20:32:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.224+0000 7f9faa800640 1 -- 172.21.15.102:0/1974450649 wait complete. 2024-04-26T20:32:33.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:32:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.446+0000 7fb089200640 1 -- 172.21.15.102:0/798092757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb084102970 msgr2=0x7fb084102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.446+0000 7fb089200640 1 --2- 172.21.15.102:0/798092757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb084102970 0x7fb084102d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb074009900 tx=0x7fb07402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.447+0000 7fb089200640 1 -- 172.21.15.102:0/798092757 shutdown_connections 2024-04-26T20:32:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.447+0000 7fb089200640 1 --2- 172.21.15.102:0/798092757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb084104520 0x7fb08410adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.447+0000 7fb089200640 1 --2- 172.21.15.102:0/798092757 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb084103b60 0x7fb084103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.447+0000 7fb089200640 1 --2- 172.21.15.102:0/798092757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb084102970 0x7fb084102d90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.447+0000 7fb089200640 1 -- 172.21.15.102:0/798092757 >> 172.21.15.102:0/798092757 conn(0x7fb0840fdee0 msgr2=0x7fb084100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.447+0000 7fb089200640 1 -- 172.21.15.102:0/798092757 shutdown_connections 2024-04-26T20:32:33.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.447+0000 7fb089200640 1 -- 172.21.15.102:0/798092757 wait complete. 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.448+0000 7fb089200640 1 Processor -- start 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.448+0000 7fb089200640 1 -- start start 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.448+0000 7fb089200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb084102970 0x7fb084198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.448+0000 7fb089200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb084103b60 0x7fb084199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.448+0000 7fb089200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb084104520 0x7fb0841a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb089200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb08410e4e0 con 0x7fb084104520 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb089200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb08410e360 con 0x7fb084103b60 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb089200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb08410e660 con 0x7fb084102970 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb081e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb084102970 0x7fb084198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb081e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb084102970 0x7fb084198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:47772/0 (socket says 172.21.15.102:47772) 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb081e00640 1 -- 172.21.15.102:0/2659560012 learned_addr learned my addr 172.21.15.102:0/2659560012 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb079400640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb084103b60 0x7fb084199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb082800640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb084104520 0x7fb0841a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb081e00640 1 -- 172.21.15.102:0/2659560012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb084103b60 msgr2=0x7fb084199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb081e00640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb084103b60 0x7fb084199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb081e00640 1 -- 172.21.15.102:0/2659560012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb084104520 msgr2=0x7fb0841a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb081e00640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb084104520 0x7fb0841a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb081e00640 1 -- 172.21.15.102:0/2659560012 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0841a07f0 con 0x7fb084102970 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.449+0000 7fb081e00640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb084102970 0x7fb084198bd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb074031500 tx=0x7fb074003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.450+0000 7fb07b400640 1 -- 172.21.15.102:0/2659560012 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07403f070 con 0x7fb084102970 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.450+0000 7fb07b400640 1 -- 172.21.15.102:0/2659560012 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb074031c50 con 0x7fb084102970 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.450+0000 7fb07b400640 1 -- 172.21.15.102:0/2659560012 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb074035030 con 0x7fb084102970 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.450+0000 7fb089200640 1 -- 172.21.15.102:0/2659560012 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0841a0a80 con 0x7fb084102970 2024-04-26T20:32:33.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.450+0000 7fb089200640 1 -- 172.21.15.102:0/2659560012 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0841a0f20 con 0x7fb084102970 2024-04-26T20:32:33.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.451+0000 7fb089200640 1 -- 172.21.15.102:0/2659560012 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb084109c90 con 0x7fb084102970 2024-04-26T20:32:33.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.453+0000 7fb07b400640 1 -- 172.21.15.102:0/2659560012 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fb074042070 con 0x7fb084102970 2024-04-26T20:32:33.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.454+0000 7fb07b400640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb0680a3c30 0x7fb0680a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.454+0000 7fb07b400640 1 -- 172.21.15.102:0/2659560012 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb0740f0d00 con 0x7fb084102970 2024-04-26T20:32:33.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.454+0000 7fb079400640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb0680a3c30 0x7fb0680a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.455+0000 7fb079400640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb0680a3c30 0x7fb0680a60f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb08419a0e0 tx=0x7fb06c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:33.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.459+0000 7fb07b400640 1 -- 172.21.15.102:0/2659560012 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb0740bcdf0 con 0x7fb084102970 2024-04-26T20:32:33.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:33 smithi102 ceph-mon[28094]: pgmap v321: 129 pgs: 12 creating+peering, 101 active+clean, 16 unknown; 579 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 978 B/s wr, 7 op/s 2024-04-26T20:32:33.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:33 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3530694318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:33.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:33 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1974450649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:33.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.664+0000 7fb089200640 1 -- 172.21.15.102:0/2659560012 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb084063560 con 0x7fb084102970 2024-04-26T20:32:33.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.666+0000 7fb07b400640 1 -- 172.21.15.102:0/2659560012 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24164 (secure 0 0 0) 0x7fb0740c10f0 con 0x7fb084102970 2024-04-26T20:32:33.667 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:33.667 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:32:30.827511+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:32:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.668+0000 7fb089200640 1 -- 172.21.15.102:0/2659560012 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb0680a3c30 msgr2=0x7fb0680a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.668+0000 7fb089200640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb0680a3c30 0x7fb0680a60f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb08419a0e0 tx=0x7fb06c005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.668+0000 7fb089200640 1 -- 172.21.15.102:0/2659560012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb084102970 msgr2=0x7fb084198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.668+0000 7fb089200640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb084102970 0x7fb084198bd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb074031500 tx=0x7fb074003c10 comp rx=0 tx=0).stop 2024-04-26T20:32:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.669+0000 7fb089200640 1 -- 172.21.15.102:0/2659560012 shutdown_connections 2024-04-26T20:32:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.669+0000 7fb089200640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb0680a3c30 0x7fb0680a60f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.669+0000 7fb089200640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb084104520 0x7fb0841a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.669+0000 7fb089200640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb084103b60 0x7fb084199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.669+0000 7fb089200640 1 --2- 172.21.15.102:0/2659560012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb084102970 0x7fb084198bd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.669+0000 7fb089200640 1 -- 172.21.15.102:0/2659560012 >> 172.21.15.102:0/2659560012 conn(0x7fb0840fdee0 msgr2=0x7fb084106e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:33.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.669+0000 7fb089200640 1 -- 172.21.15.102:0/2659560012 shutdown_connections 2024-04-26T20:32:33.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.669+0000 7fb089200640 1 -- 172.21.15.102:0/2659560012 wait complete. 2024-04-26T20:32:33.690 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T20:32:25.769708+0000', 'modified': '2024-04-26T20:32:30.827828+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.112:6839/2318597180', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 2318597180}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 2318597180}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24593, 'qdb_cluster': [24593]} 2024-04-26T20:32:33.691 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T20:32:33.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T20:32:33.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:33 smithi112 ceph-mon[32769]: pgmap v321: 129 pgs: 12 creating+peering, 101 active+clean, 16 unknown; 579 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 978 B/s wr, 7 op/s 2024-04-26T20:32:33.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:33 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3530694318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:33.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:33 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1974450649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:33 smithi130 ceph-mon[32704]: pgmap v321: 129 pgs: 12 creating+peering, 101 active+clean, 16 unknown; 579 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 978 B/s wr, 7 op/s 2024-04-26T20:32:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:33 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3530694318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:33.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:33 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1974450649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:33.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.892+0000 7f9b49400640 1 -- 172.21.15.102:0/622989791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b4410a940 msgr2=0x7f9b441111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.892+0000 7f9b49400640 1 --2- 172.21.15.102:0/622989791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b4410a940 0x7f9b441111d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009860 tx=0x7f9b3c02a340 comp rx=0 tx=0).stop 2024-04-26T20:32:33.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.892+0000 7f9b49400640 1 -- 172.21.15.102:0/622989791 shutdown_connections 2024-04-26T20:32:33.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.892+0000 7f9b49400640 1 --2- 172.21.15.102:0/622989791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b44109f80 0x7f9b4410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.892+0000 7f9b49400640 1 --2- 172.21.15.102:0/622989791 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b4410a940 0x7f9b441111d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.892+0000 7f9b49400640 1 --2- 172.21.15.102:0/622989791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b44108d90 0x7f9b441091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.892+0000 7f9b49400640 1 -- 172.21.15.102:0/622989791 >> 172.21.15.102:0/622989791 conn(0x7f9b44077d30 msgr2=0x7f9b4407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.892+0000 7f9b49400640 1 -- 172.21.15.102:0/622989791 shutdown_connections 2024-04-26T20:32:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.892+0000 7f9b49400640 1 -- 172.21.15.102:0/622989791 wait complete. 2024-04-26T20:32:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.893+0000 7f9b49400640 1 Processor -- start 2024-04-26T20:32:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.893+0000 7f9b49400640 1 -- start start 2024-04-26T20:32:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.893+0000 7f9b49400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b44108d90 0x7f9b4419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b49400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b44109f80 0x7f9b4419f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b49400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b4410a940 0x7f9b441a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b49400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441148c0 con 0x7f9b44109f80 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b49400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9b44114740 con 0x7f9b44108d90 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b49400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9b44114a40 con 0x7f9b4410a940 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b3b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b44109f80 0x7f9b4419f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b3b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b44109f80 0x7f9b4419f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44906/0 (socket says 172.21.15.102:44906) 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b48a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b4410a940 0x7f9b441a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b3b400640 1 -- 172.21.15.102:0/3046616968 learned_addr learned my addr 172.21.15.102:0/3046616968 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b43e00640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b44108d90 0x7f9b4419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b48a00640 1 -- 172.21.15.102:0/3046616968 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b44108d90 msgr2=0x7f9b4419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b48a00640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b44108d90 0x7f9b4419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b48a00640 1 -- 172.21.15.102:0/3046616968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b44109f80 msgr2=0x7f9b4419f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b48a00640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b44109f80 0x7f9b4419f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b48a00640 1 -- 172.21.15.102:0/3046616968 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b441a6b20 con 0x7f9b4410a940 2024-04-26T20:32:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.894+0000 7f9b3b400640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b44109f80 0x7f9b4419f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.895+0000 7f9b48a00640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b4410a940 0x7f9b441a64b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c02a930 tx=0x7f9b3c032dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.895+0000 7f9b41600640 1 -- 172.21.15.102:0/3046616968 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c040030 con 0x7f9b4410a940 2024-04-26T20:32:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.895+0000 7f9b49400640 1 -- 172.21.15.102:0/3046616968 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b441a6db0 con 0x7f9b4410a940 2024-04-26T20:32:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.895+0000 7f9b41600640 1 -- 172.21.15.102:0/3046616968 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c030bd0 con 0x7f9b4410a940 2024-04-26T20:32:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.895+0000 7f9b41600640 1 -- 172.21.15.102:0/3046616968 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c037040 con 0x7f9b4410a940 2024-04-26T20:32:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.895+0000 7f9b49400640 1 -- 172.21.15.102:0/3046616968 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b441a7260 con 0x7f9b4410a940 2024-04-26T20:32:33.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.899+0000 7f9b4d87c740 1 Processor -- start 2024-04-26T20:32:33.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b41600640 1 -- 172.21.15.102:0/3046616968 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9b3c043070 con 0x7f9b4410a940 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b4d87c740 1 -- start start 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6a2c0 0x557f1fa6a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa6f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 0x557f1fa722b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557f1f9ce600 con 0x557f1fa6acb0 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f1f96b100 con 0x557f1fa6fe20 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557f1f9ca5c0 con 0x557f1fa6a2c0 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b43e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa6f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b43e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa6f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44908/0 (socket says 172.21.15.102:44908) 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.900+0000 7f9b43e00640 1 -- 172.21.15.102:0/3977849872 learned_addr learned my addr 172.21.15.102:0/3977849872 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b3b400640 1 --2- 172.21.15.102:0/3977849872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6a2c0 0x557f1fa6a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b3b400640 1 --2- 172.21.15.102:0/3977849872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 0x557f1fa722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b43e00640 1 -- 172.21.15.102:0/3977849872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6a2c0 msgr2=0x557f1fa6a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b43e00640 1 --2- 172.21.15.102:0/3977849872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6a2c0 0x557f1fa6a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b43e00640 1 -- 172.21.15.102:0/3977849872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 msgr2=0x557f1fa722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b43e00640 1 --2- 172.21.15.102:0/3977849872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 0x557f1fa722b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b43e00640 1 -- 172.21.15.102:0/3977849872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f1f6e02d0 con 0x557f1fa6acb0 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b3b400640 1 --2- 172.21.15.102:0/3977849872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 0x557f1fa722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b41600640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9b280a3c30 0x7f9b280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b43e00640 1 --2- 172.21.15.102:0/3977849872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa6f7b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b34011470 tx=0x7f9b34011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f0958f82160a32 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b41600640 1 -- 172.21.15.102:0/3046616968 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9b3c0efec0 con 0x7f9b4410a940 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b3b400640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9b280a3c30 0x7f9b280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.901+0000 7f9b39200640 1 -- 172.21.15.102:0/3977849872 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34003970 con 0x557f1fa6acb0 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.902+0000 7f9b39200640 1 -- 172.21.15.102:0/3977849872 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b34004020 con 0x557f1fa6acb0 2024-04-26T20:32:33.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.902+0000 7f9b39200640 1 -- 172.21.15.102:0/3977849872 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3400f8d0 con 0x557f1fa6acb0 2024-04-26T20:32:33.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.902+0000 7f9b4d87c740 1 -- 172.21.15.102:0/3977849872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 msgr2=0x557f1fa6f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.902+0000 7f9b3b400640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9b280a3c30 0x7f9b280a60f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x557f1fa6b510 tx=0x7f9b2c013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:33.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.902+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/3977849872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa6f7b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b34011470 tx=0x7f9b34011930 comp rx=0 tx=0).stop 2024-04-26T20:32:33.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.902+0000 7f9b4d87c740 1 -- 172.21.15.102:0/3977849872 shutdown_connections 2024-04-26T20:32:33.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.902+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/3977849872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa6f7b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.902+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/3977849872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 0x557f1fa722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.902+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/3977849872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6a2c0 0x557f1fa6a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.902+0000 7f9b4d87c740 1 -- 172.21.15.102:0/3977849872 >> 172.21.15.102:0/3977849872 conn(0x557f1f9fed10 msgr2=0x557f1fa65d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:33.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.903+0000 7f9b4d87c740 1 -- 172.21.15.102:0/3977849872 shutdown_connections 2024-04-26T20:32:33.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.903+0000 7f9b4d87c740 1 -- 172.21.15.102:0/3977849872 wait complete. 2024-04-26T20:32:33.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.903+0000 7f9b4d87c740 1 -- start start 2024-04-26T20:32:33.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.903+0000 7f9b4d87c740 1 Processor -- start 2024-04-26T20:32:33.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 0x557f1fa7bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557f1fa72dd0 con 0x557f1fa6acb0 2024-04-26T20:32:33.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f1f955f70 con 0x557f1fa6a2c0 2024-04-26T20:32:33.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557f1fa72f50 con 0x557f1fa6fe20 2024-04-26T20:32:33.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b4d87c740 10 client.0 _ll_register_callbacks cb 0x557f1f9c4480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:32:33.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b3b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b48a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 0x557f1fa7bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b3b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44922/0 (socket says 172.21.15.102:44922) 2024-04-26T20:32:33.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b3b400640 1 -- 172.21.15.102:0/4100009538 learned_addr learned my addr 172.21.15.102:0/4100009538 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:33.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b43e00640 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b48a00640 1 -- 172.21.15.102:0/4100009538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 msgr2=0x557f1fa746f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:32:33.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b48a00640 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b48a00640 1 -- 172.21.15.102:0/4100009538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 msgr2=0x557f1fa74c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:32:33.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b48a00640 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b48a00640 1 -- 172.21.15.102:0/4100009538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f1f7aa6b0 con 0x557f1fa6fe20 2024-04-26T20:32:33.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b43e00640 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:33.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.904+0000 7f9b3b400640 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:33.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.905+0000 7f9b48a00640 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 0x557f1fa7bc70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0bbe90 tx=0x7f9b3c0328c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1df85ec4c277f82 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:33.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.905+0000 7f9b31600640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-26T20:32:33.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.905+0000 7f9b31600640 1 -- 172.21.15.102:0/4100009538 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0c02b0 con 0x557f1fa6fe20 2024-04-26T20:32:33.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.905+0000 7f9b4d87c740 1 -- 172.21.15.102:0/4100009538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b34004020 con 0x557f1fa6fe20 2024-04-26T20:32:33.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.905+0000 7f9b31600640 1 -- 172.21.15.102:0/4100009538 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9b3c0ba200 con 0x557f1fa6fe20 2024-04-26T20:32:33.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.905+0000 7f9b31600640 1 -- 172.21.15.102:0/4100009538 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f9b3c0c4b70 con 0x557f1fa6fe20 2024-04-26T20:32:33.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.905+0000 7f9b4d87c740 10 client.24697 fetch_fsmap learned FSMap version 16 2024-04-26T20:32:33.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.905+0000 7f9b4d87c740 1 -- 172.21.15.102:0/4100009538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x557f1fa72dd0 con 0x557f1fa6fe20 2024-04-26T20:32:33.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.905+0000 7f9b31600640 1 -- 172.21.15.102:0/4100009538 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b34004020 con 0x557f1fa6fe20 2024-04-26T20:32:33.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.905+0000 7f9b31600640 1 -- 172.21.15.102:0/4100009538 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7f9b3c0edbe0 con 0x557f1fa6fe20 2024-04-26T20:32:33.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.906+0000 7f9b4d87c740 10 client.24697 fetch_fsmap finished waiting for FSMap version 16 2024-04-26T20:32:33.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.906+0000 7f9b4d87c740 10 client.24697 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:32:33.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.906+0000 7f9b4d87c740 20 client.24697 populate_metadata read hostname 'smithi102' 2024-04-26T20:32:33.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.906+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f1fa81440 0x557f1fa828b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.906+0000 7f9b4d87c740 4 client.24697 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:32:33.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.906+0000 7f9b4d87c740 1 -- 172.21.15.102:0/4100009538 --> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f1f9dc210 con 0x557f1fa81440 2024-04-26T20:32:33.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.906+0000 7f9b43e00640 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f1fa81440 0x557f1fa828b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.907+0000 7f9b43e00640 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f1fa81440 0x557f1fa828b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50826c910342ee18 server_cookie=c23ae62548cecd6a in_seq=0 out_seq=0 2024-04-26T20:32:33.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.907+0000 7f9b31600640 10 client.24697 ms_handle_connect on v2:172.21.15.112:6838/2318597180 2024-04-26T20:32:33.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.907+0000 7f9b31600640 1 -- 172.21.15.102:0/4100009538 <== mds.0 v2:172.21.15.112:6838/2318597180 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f1f9dc210 con 0x557f1fa81440 2024-04-26T20:32:33.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.907+0000 7f9b31600640 10 client.24697 handle_command_reply: tid=0 2024-04-26T20:32:33.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.907+0000 7f9b4d87c740 1 client.24697 shutdown 2024-04-26T20:32:33.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 1 -- 172.21.15.102:0/4100009538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 msgr2=0x557f1fa7bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 0x557f1fa7bc70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0bbe90 tx=0x7f9b3c0328c0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 1 -- 172.21.15.102:0/4100009538 shutdown_connections 2024-04-26T20:32:33.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f1fa81440 0x557f1fa828b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 0x557f1fa7bc70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/4100009538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 1 -- 172.21.15.102:0/4100009538 >> 172.21.15.102:0/4100009538 conn(0x557f1f9fed10 msgr2=0x557f1fa64f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:33.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b31600640 0 client.24697 ms_handle_reset on v2:172.21.15.112:6838/2318597180 2024-04-26T20:32:33.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 1 -- 172.21.15.102:0/4100009538 shutdown_connections 2024-04-26T20:32:33.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 1 -- 172.21.15.102:0/4100009538 wait complete. 2024-04-26T20:32:33.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.908+0000 7f9b4d87c740 20 client.24697 trim_cache size 0 max 16384 2024-04-26T20:32:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b4d87c740 1 Processor -- start 2024-04-26T20:32:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b4d87c740 1 -- start start 2024-04-26T20:32:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 0x557f1fa7bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557f1fa80bf0 con 0x557f1fa6acb0 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f1fa812c0 con 0x557f1fa6a2c0 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557f1fa81990 con 0x557f1fa6fe20 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b43e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b43e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:55126/0 (socket says 172.21.15.102:55126) 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b43e00640 1 -- 172.21.15.102:0/3546656841 learned_addr learned my addr 172.21.15.102:0/3546656841 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b3b400640 1 --2- 172.21.15.102:0/3546656841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.928+0000 7f9b48a00640 1 --2- 172.21.15.102:0/3546656841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 0x557f1fa7bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.929+0000 7f9b43e00640 1 -- 172.21.15.102:0/3546656841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 msgr2=0x557f1fa7bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.929+0000 7f9b43e00640 1 --2- 172.21.15.102:0/3546656841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 0x557f1fa7bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.929+0000 7f9b43e00640 1 -- 172.21.15.102:0/3546656841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 msgr2=0x557f1fa74c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.929+0000 7f9b43e00640 1 --2- 172.21.15.102:0/3546656841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.929+0000 7f9b43e00640 1 -- 172.21.15.102:0/3546656841 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f1fa6f340 con 0x557f1fa6a2c0 2024-04-26T20:32:33.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.929+0000 7f9b3b400640 1 --2- 172.21.15.102:0/3546656841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.929+0000 7f9b43e00640 1 --2- 172.21.15.102:0/3546656841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9b34011470 tx=0x7f9b34013fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a76cd84c182822e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.929+0000 7f9b33e00640 1 -- 172.21.15.102:0/3546656841 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34009b30 con 0x557f1fa6a2c0 2024-04-26T20:32:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.929+0000 7f9b33e00640 1 -- 172.21.15.102:0/3546656841 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b34003e20 con 0x557f1fa6a2c0 2024-04-26T20:32:33.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.929+0000 7f9b33e00640 1 -- 172.21.15.102:0/3546656841 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3400f7e0 con 0x557f1fa6a2c0 2024-04-26T20:32:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.930+0000 7f9b4d87c740 1 -- 172.21.15.102:0/3546656841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 msgr2=0x557f1fa746f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.930+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/3546656841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9b34011470 tx=0x7f9b34013fd0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.930+0000 7f9b4d87c740 1 -- 172.21.15.102:0/3546656841 shutdown_connections 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.930+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/3546656841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6acb0 0x557f1fa74c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.930+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/3546656841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6fe20 0x557f1fa7bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.930+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/3546656841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6a2c0 0x557f1fa746f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.930+0000 7f9b4d87c740 1 -- 172.21.15.102:0/3546656841 >> 172.21.15.102:0/3546656841 conn(0x557f1f9fed10 msgr2=0x557f1fa60df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.930+0000 7f9b4d87c740 1 -- 172.21.15.102:0/3546656841 shutdown_connections 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.930+0000 7f9b4d87c740 1 -- 172.21.15.102:0/3546656841 wait complete. 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.930+0000 7f9b4d87c740 1 -- start start 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b4d87c740 1 Processor -- start 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6a2c0 0x557f1fa773b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6acb0 0x557f1fa778f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b4d87c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 0x557f1fa7a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557f1fa82730 con 0x557f1fa6a2c0 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f1fa82060 con 0x557f1fa6fe20 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b4d87c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557f1fa82e00 con 0x557f1fa6acb0 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b4d87c740 10 client.0 _ll_register_callbacks cb 0x7f9b3c005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b43e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6a2c0 0x557f1fa773b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b48a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 0x557f1fa7a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b43e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6a2c0 0x557f1fa773b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44942/0 (socket says 172.21.15.102:44942) 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b43e00640 1 -- 172.21.15.102:0/2081638751 learned_addr learned my addr 172.21.15.102:0/2081638751 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.931+0000 7f9b3b400640 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6acb0 0x557f1fa778f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b48a00640 1 -- 172.21.15.102:0/2081638751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6acb0 msgr2=0x557f1fa778f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b48a00640 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6acb0 0x557f1fa778f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b48a00640 1 -- 172.21.15.102:0/2081638751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6a2c0 msgr2=0x557f1fa773b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b48a00640 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6a2c0 0x557f1fa773b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b48a00640 1 -- 172.21.15.102:0/2081638751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f1fa7af00 con 0x557f1fa6fe20 2024-04-26T20:32:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b43e00640 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6a2c0 0x557f1fa773b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b48a00640 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 0x557f1fa7a910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0b88d0 tx=0x7f9b3c005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2744dcb096d6c819 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b31600640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b31600640 1 -- 172.21.15.102:0/2081638751 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0bba40 con 0x557f1fa6fe20 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b4d87c740 1 -- 172.21.15.102:0/2081638751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b34003e20 con 0x557f1fa6fe20 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b31600640 1 -- 172.21.15.102:0/2081638751 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9b3c0ba200 con 0x557f1fa6fe20 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b31600640 1 -- 172.21.15.102:0/2081638751 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f9b3c0c02b0 con 0x557f1fa6fe20 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b4d87c740 10 client.24707 fetch_fsmap learned FSMap version 16 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.932+0000 7f9b4d87c740 1 -- 172.21.15.102:0/2081638751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x557f1fae73f0 con 0x557f1fa6fe20 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.933+0000 7f9b31600640 1 -- 172.21.15.102:0/2081638751 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b34003e20 con 0x557f1fa6fe20 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.933+0000 7f9b31600640 1 -- 172.21.15.102:0/2081638751 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7f9b3c004040 con 0x557f1fa6fe20 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.933+0000 7f9b4d87c740 10 client.24707 fetch_fsmap finished waiting for FSMap version 16 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.933+0000 7f9b4d87c740 10 client.24707 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.933+0000 7f9b4d87c740 20 client.24707 populate_metadata read hostname 'smithi102' 2024-04-26T20:32:33.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.933+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f1faebbf0 0x557f1faea900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.933+0000 7f9b4d87c740 4 client.24707 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.933+0000 7f9b4d87c740 1 -- 172.21.15.102:0/2081638751 --> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] -- command(tid 0: {"prefix": "status"}) -- 0x557f1f9d6080 con 0x557f1faebbf0 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.933+0000 7f9b43e00640 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f1faebbf0 0x557f1faea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b43e00640 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f1faebbf0 0x557f1faea900 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73c5e5133b6ef123 server_cookie=515a34b5ad0d7702 in_seq=0 out_seq=0 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b31600640 10 client.24707 ms_handle_connect on v2:172.21.15.112:6838/2318597180 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b31600640 1 -- 172.21.15.102:0/2081638751 <== mds.0 v2:172.21.15.112:6838/2318597180 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x557f1f9d6080 con 0x557f1faebbf0 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b31600640 10 client.24707 handle_command_reply: tid=0 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 client.24707 shutdown 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 -- 172.21.15.102:0/2081638751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 msgr2=0x557f1fa7a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 0x557f1fa7a910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0b88d0 tx=0x7f9b3c005280 comp rx=0 tx=0).stop 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 -- 172.21.15.102:0/2081638751 shutdown_connections 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f1faebbf0 0x557f1faea900 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f1fa6fe20 0x557f1fa7a910 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b31600640 0 client.24707 ms_handle_reset on v2:172.21.15.112:6838/2318597180 2024-04-26T20:32:33.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f1fa6acb0 0x557f1fa778f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 --2- 172.21.15.102:0/2081638751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f1fa6a2c0 0x557f1fa773b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 -- 172.21.15.102:0/2081638751 >> 172.21.15.102:0/2081638751 conn(0x557f1f9fed10 msgr2=0x557f1fa746f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:33.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 -- 172.21.15.102:0/2081638751 shutdown_connections 2024-04-26T20:32:33.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.934+0000 7f9b4d87c740 1 -- 172.21.15.102:0/2081638751 wait complete. 2024-04-26T20:32:33.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b4d87c740 20 client.24707 trim_cache size 0 max 16384 2024-04-26T20:32:33.941 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24593, 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 7.6277707850000001, 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 16, 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 89, 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 13.985500904 2024-04-26T20:32:33.942 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:32:33.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 -- 172.21.15.102:0/3046616968 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9b280a3c30 msgr2=0x7f9b280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9b280a3c30 0x7f9b280a60f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x557f1fa6b510 tx=0x7f9b2c013040 comp rx=0 tx=0).stop 2024-04-26T20:32:33.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 -- 172.21.15.102:0/3046616968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b4410a940 msgr2=0x7f9b441a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:33.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b4410a940 0x7f9b441a64b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c02a930 tx=0x7f9b3c032dc0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 -- 172.21.15.102:0/3046616968 shutdown_connections 2024-04-26T20:32:33.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9b280a3c30 0x7f9b280a60f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b44109f80 0x7f9b4419f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b4410a940 0x7f9b441a64b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 --2- 172.21.15.102:0/3046616968 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b44108d90 0x7f9b4419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:33.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 -- 172.21.15.102:0/3046616968 >> 172.21.15.102:0/3046616968 conn(0x7f9b44077d30 msgr2=0x7f9b4410d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:33.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 -- 172.21.15.102:0/3046616968 shutdown_connections 2024-04-26T20:32:33.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:33.935+0000 7f9b49400640 1 -- 172.21.15.102:0/3046616968 wait complete. 2024-04-26T20:32:33.954 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T20:32:33.954 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:32:33.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T20:32:34.395 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:32:34.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:34 smithi102 ceph-mon[28094]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:32:34.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:34 smithi102 ceph-mon[28094]: Cluster is now healthy 2024-04-26T20:32:34.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:34 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2659560012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:34.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:34 smithi112 ceph-mon[32769]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:32:34.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:34 smithi112 ceph-mon[32769]: Cluster is now healthy 2024-04-26T20:32:34.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:34 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2659560012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:34.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:34 smithi130 ceph-mon[32704]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T20:32:34.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:34 smithi130 ceph-mon[32704]: Cluster is now healthy 2024-04-26T20:32:34.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:34 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2659560012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:35.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:35 smithi102 ceph-mon[28094]: pgmap v322: 129 pgs: 12 creating+peering, 117 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 5 op/s 2024-04-26T20:32:35.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:35 smithi112 ceph-mon[32769]: pgmap v322: 129 pgs: 12 creating+peering, 117 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 5 op/s 2024-04-26T20:32:35.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:35 smithi130 ceph-mon[32704]: pgmap v322: 129 pgs: 12 creating+peering, 117 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 5 op/s 2024-04-26T20:32:36.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.760+0000 7fc51d800700 1 -- 172.21.15.102:0/2748095923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc518103470 msgr2=0x7fc5181038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:36.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.760+0000 7fc51d800700 1 --2- 172.21.15.102:0/2748095923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc518103470 0x7fc5181038f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc5000098f0 tx=0x7fc500009c00 comp rx=0 tx=0).stop 2024-04-26T20:32:36.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.761+0000 7fc51d800700 1 -- 172.21.15.102:0/2748095923 shutdown_connections 2024-04-26T20:32:36.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.761+0000 7fc51d800700 1 --2- 172.21.15.102:0/2748095923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc518103e30 0x7fc518110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:36.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.761+0000 7fc51d800700 1 --2- 172.21.15.102:0/2748095923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5181094c0 0x7fc5181098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:36.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.761+0000 7fc51d800700 1 --2- 172.21.15.102:0/2748095923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc518103470 0x7fc5181038f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:36.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.761+0000 7fc51d800700 1 -- 172.21.15.102:0/2748095923 >> 172.21.15.102:0/2748095923 conn(0x7fc51807a580 msgr2=0x7fc51807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:36.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.761+0000 7fc51d800700 1 -- 172.21.15.102:0/2748095923 shutdown_connections 2024-04-26T20:32:36.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.761+0000 7fc51d800700 1 -- 172.21.15.102:0/2748095923 wait complete. 2024-04-26T20:32:36.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.762+0000 7fc51d800700 1 Processor -- start 2024-04-26T20:32:36.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.762+0000 7fc51d800700 1 -- start start 2024-04-26T20:32:36.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc51d800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc518103470 0x7fc51819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:36.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc51d800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc518103e30 0x7fc51819fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:36.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc51d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5181094c0 0x7fc5181999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:36.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc51d800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc518113640 con 0x7fc5181094c0 2024-04-26T20:32:36.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc51d800700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181134c0 con 0x7fc518103470 2024-04-26T20:32:36.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc51d800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc5181137c0 con 0x7fc518103e30 2024-04-26T20:32:36.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc50fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc518103e30 0x7fc51819fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:36.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc50fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc518103e30 0x7fc51819fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:58460/0 (socket says 172.21.15.102:58460) 2024-04-26T20:32:36.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc50fe00700 1 -- 172.21.15.102:0/1595145941 learned_addr learned my addr 172.21.15.102:0/1595145941 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:36.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc517400700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5181094c0 0x7fc5181999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc516a00700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc518103470 0x7fc51819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc50fe00700 1 -- 172.21.15.102:0/1595145941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc518103470 msgr2=0x7fc51819f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc50fe00700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc518103470 0x7fc51819f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc50fe00700 1 -- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5181094c0 msgr2=0x7fc5181999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc50fe00700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5181094c0 0x7fc5181999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc50fe00700 1 -- 172.21.15.102:0/1595145941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51819a140 con 0x7fc518103e30 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc516a00700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc518103470 0x7fc51819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.763+0000 7fc517400700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5181094c0 0x7fc5181999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.764+0000 7fc50fe00700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc518103e30 0x7fc51819fd60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc500009230 tx=0x7fc50000fb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.764+0000 7fc50f400700 1 -- 172.21.15.102:0/1595145941 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500005220 con 0x7fc518103e30 2024-04-26T20:32:36.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.764+0000 7fc50f400700 1 -- 172.21.15.102:0/1595145941 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc50000b560 con 0x7fc518103e30 2024-04-26T20:32:36.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.764+0000 7fc50f400700 1 -- 172.21.15.102:0/1595145941 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500003b40 con 0x7fc518103e30 2024-04-26T20:32:36.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.764+0000 7fc51d800700 1 -- 172.21.15.102:0/1595145941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc51819a3d0 con 0x7fc518103e30 2024-04-26T20:32:36.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.764+0000 7fc51d800700 1 -- 172.21.15.102:0/1595145941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5181a66f0 con 0x7fc518103e30 2024-04-26T20:32:36.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.765+0000 7fc51d800700 1 -- 172.21.15.102:0/1595145941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc51810b930 con 0x7fc518103e30 2024-04-26T20:32:36.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.767+0000 7fc50f400700 1 -- 172.21.15.102:0/1595145941 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fc50000b710 con 0x7fc518103e30 2024-04-26T20:32:36.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.769+0000 7fc50f400700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4fc0a3cc0 0x7fc4fc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:36.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.769+0000 7fc50f400700 1 -- 172.21.15.102:0/1595145941 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc5000d1820 con 0x7fc518103e30 2024-04-26T20:32:36.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.769+0000 7fc516a00700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4fc0a3cc0 0x7fc4fc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:36.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.770+0000 7fc516a00700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4fc0a3cc0 0x7fc4fc0a6180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc51819ab20 tx=0x7fc508009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:36.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:36.771+0000 7fc50f400700 1 -- 172.21.15.102:0/1595145941 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc50005f060 con 0x7fc518103e30 2024-04-26T20:32:37.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.049+0000 7fc51d800700 1 -- 172.21.15.102:0/1595145941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc51819af10 con 0x7fc518103e30 2024-04-26T20:32:37.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.050+0000 7fc50f400700 1 -- 172.21.15.102:0/1595145941 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7fc500013030 con 0x7fc518103e30 2024-04-26T20:32:37.052 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T20:32:37.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.053+0000 7fc51d800700 1 -- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4fc0a3cc0 msgr2=0x7fc4fc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.053+0000 7fc51d800700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4fc0a3cc0 0x7fc4fc0a6180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc51819ab20 tx=0x7fc508009310 comp rx=0 tx=0).stop 2024-04-26T20:32:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.054+0000 7fc51d800700 1 -- 172.21.15.102:0/1595145941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc518103e30 msgr2=0x7fc51819fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.054+0000 7fc51d800700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc518103e30 0x7fc51819fd60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc500009230 tx=0x7fc50000fb20 comp rx=0 tx=0).stop 2024-04-26T20:32:37.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.054+0000 7fc51d800700 1 -- 172.21.15.102:0/1595145941 shutdown_connections 2024-04-26T20:32:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.054+0000 7fc51d800700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4fc0a3cc0 0x7fc4fc0a6180 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.054+0000 7fc51d800700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5181094c0 0x7fc5181999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.054+0000 7fc51d800700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc518103470 0x7fc51819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.054+0000 7fc51d800700 1 --2- 172.21.15.102:0/1595145941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc518103e30 0x7fc51819fd60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.054+0000 7fc51d800700 1 -- 172.21.15.102:0/1595145941 >> 172.21.15.102:0/1595145941 conn(0x7fc51807a580 msgr2=0x7fc518102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.054+0000 7fc51d800700 1 -- 172.21.15.102:0/1595145941 shutdown_connections 2024-04-26T20:32:37.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:37.054+0000 7fc51d800700 1 -- 172.21.15.102:0/1595145941 wait complete. 2024-04-26T20:32:37.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:37 smithi130 ceph-mon[32704]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 285 B/s wr, 3 op/s 2024-04-26T20:32:37.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:37 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1595145941' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:32:37.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:37 smithi102 ceph-mon[28094]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 285 B/s wr, 3 op/s 2024-04-26T20:32:37.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:37 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1595145941' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:32:38.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:37 smithi112 ceph-mon[32769]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 285 B/s wr, 3 op/s 2024-04-26T20:32:38.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:37 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1595145941' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:32:38.132 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T20:32:38.132 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T20:32:38.132 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-04-26T20:32:38.132 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-26T20:32:38.132 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:32:38.132 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T20:32:38.132 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T20:32:38.132 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T20:32:38.132 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T20:32:38.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:38.133 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-04-26T20:32:38.190 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T20:32:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T20:32:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T20:32:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:32:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-04-26T20:32:38.190 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42094sec preferred_lft 42094sec 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-04-26T20:32:38.191 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:32:38.193 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T20:32:38.193 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:32:38.193 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-26T20:32:38.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add name ceph-brx type bridge 2024-04-26T20:32:38.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr flush dev ceph-brx 2024-04-26T20:32:38.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set ceph-brx up 2024-04-26T20:32:38.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T20:32:38.193 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-26T20:32:38.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:32:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:32:38.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:38.695 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T20:32:38.762 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-04-26T20:32:38.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:38.766 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-04-26T20:32:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-04-26T20:32:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T20:32:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-26T20:32:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-26T20:32:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T20:32:38.821 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:32:38.821 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-26T20:32:38.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-26T20:32:38.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-26T20:32:38.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-26T20:32:38.822 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-26T20:32:38.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:32:39.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:32:39.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:39.599 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-04-26T20:32:39.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:39.651 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2024-04-26T20:32:39.703 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:32:39.703 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-26T20:32:39.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:32:39.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-26T20:32:39.703 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-26T20:32:39.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:32:39.834 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:39 smithi102 ceph-mon[28094]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:39.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:39 smithi130 ceph-mon[32704]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:39.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:32:39.887 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-26T20:32:39.887 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:32:39.887 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-26T20:32:39.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-26T20:32:39.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T20:32:39.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-26T20:32:39.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-26T20:32:39.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-26T20:32:39.888 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-26T20:32:39.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:32:40.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:39 smithi112 ceph-mon[32769]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:40.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:32:40.175 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:32:40.175 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-26T20:32:40.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-04-26T20:32:40.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T20:32:40.176 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-26T20:32:40.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:32:40.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:32:40.374 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-26T20:32:40.374 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T20:32:40.375 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:32:40.421 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T20:32:40.421 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T20:32:40.421 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:32:40.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-04-26T20:32:40.626 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T20:32:40.673 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/selinux 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T20:32:40.674 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/76e11b296875892a0aa2ab4e8a1e60d16463fcf051392864246d7666d48e5569/merged 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/8086d303cb99c57633a5b3f93f196af07fd7a9cb7967a502d6f6da6a41f3e553/merged 2024-04-26T20:32:40.675 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/0 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/0c247f4a88d62f4ccb41963568dfe29b1900ae4a56d4e9fae329832ec255873b/merged 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/aa8c2c3c3a8486235be4d7e438d44629a337a1abcd39a5895cf771bbce8140b4/merged 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/3f9c39e2f5cbccc318c58826b57ab9b685e1cf98964ded1c951e6c73746f7138/merged 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/6864103c22b2bf5f0d9c876d60ab2375a17aa2179cec3120e3179ef019ddc903/merged 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/46a13c6d82c3bd853f6fa0c1fd699d9d2fc20e170ba17a6adb826e7f64bc8d5b/merged 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/a6f5a63f031be26e8bc984bdb5a0e548264100a8e0cae23535c32b2a93fb7840/merged 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/d74f62361bfd34525d90db67c241adb1b0d9ec8244f6dcf879d604fdbe71572a/merged 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/f6328d9302aba592870f3320e69f62da3cc8f4c28d0e9b061082069ccd56c6cc/merged 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:32:40.676 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:32:40.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:40.677 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-04-26T20:32:40.722 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-26T20:32:40.725 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-26T20:32:40.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-04-26T20:32:40.848 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T20:32:40.889 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-04-26T20:32:40.889 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-04-26T20:32:40.889 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/selinux 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-04-26T20:32:40.890 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/76e11b296875892a0aa2ab4e8a1e60d16463fcf051392864246d7666d48e5569/merged 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/8086d303cb99c57633a5b3f93f196af07fd7a9cb7967a502d6f6da6a41f3e553/merged 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/0 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T20:32:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/0c247f4a88d62f4ccb41963568dfe29b1900ae4a56d4e9fae329832ec255873b/merged 2024-04-26T20:32:40.892 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/aa8c2c3c3a8486235be4d7e438d44629a337a1abcd39a5895cf771bbce8140b4/merged 2024-04-26T20:32:40.892 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/3f9c39e2f5cbccc318c58826b57ab9b685e1cf98964ded1c951e6c73746f7138/merged 2024-04-26T20:32:40.892 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/6864103c22b2bf5f0d9c876d60ab2375a17aa2179cec3120e3179ef019ddc903/merged 2024-04-26T20:32:40.892 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/46a13c6d82c3bd853f6fa0c1fd699d9d2fc20e170ba17a6adb826e7f64bc8d5b/merged 2024-04-26T20:32:40.892 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/a6f5a63f031be26e8bc984bdb5a0e548264100a8e0cae23535c32b2a93fb7840/merged 2024-04-26T20:32:40.892 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/d74f62361bfd34525d90db67c241adb1b0d9ec8244f6dcf879d604fdbe71572a/merged 2024-04-26T20:32:40.892 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/f6328d9302aba592870f3320e69f62da3cc8f4c28d0e9b061082069ccd56c6cc/merged 2024-04-26T20:32:40.892 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-04-26T20:32:40.892 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:32:40.892 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:32:40.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:40.893 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-04-26T20:32:40.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:2024-04-26T20:32:40.966+0000 7f5c5f11d480 -1 init, newargv = 0x56282081e870 newargc=15 2024-04-26T20:32:40.968 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:ceph-fuse[71624]: starting ceph client 2024-04-26T20:32:40.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:ceph-fuse[71624]: starting fuse 2024-04-26T20:32:41.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:41 smithi130 ceph-mon[32704]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:41.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:41 smithi102 ceph-mon[28094]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:41 smithi112 ceph-mon[32769]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:42.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-04-26T20:32:42.173 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T20:32:42.232 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-04-26T20:32:42.232 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-04-26T20:32:42.232 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-04-26T20:32:42.232 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-04-26T20:32:42.232 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-04-26T20:32:42.232 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-04-26T20:32:42.232 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-04-26T20:32:42.232 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/selinux 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T20:32:42.233 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/76e11b296875892a0aa2ab4e8a1e60d16463fcf051392864246d7666d48e5569/merged 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/8086d303cb99c57633a5b3f93f196af07fd7a9cb7967a502d6f6da6a41f3e553/merged 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/0 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/0c247f4a88d62f4ccb41963568dfe29b1900ae4a56d4e9fae329832ec255873b/merged 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/aa8c2c3c3a8486235be4d7e438d44629a337a1abcd39a5895cf771bbce8140b4/merged 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/3f9c39e2f5cbccc318c58826b57ab9b685e1cf98964ded1c951e6c73746f7138/merged 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/6864103c22b2bf5f0d9c876d60ab2375a17aa2179cec3120e3179ef019ddc903/merged 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/46a13c6d82c3bd853f6fa0c1fd699d9d2fc20e170ba17a6adb826e7f64bc8d5b/merged 2024-04-26T20:32:42.234 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/a6f5a63f031be26e8bc984bdb5a0e548264100a8e0cae23535c32b2a93fb7840/merged 2024-04-26T20:32:42.235 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/d74f62361bfd34525d90db67c241adb1b0d9ec8244f6dcf879d604fdbe71572a/merged 2024-04-26T20:32:42.235 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/containers/storage/overlay/f6328d9302aba592870f3320e69f62da3cc8f4c28d0e9b061082069ccd56c6cc/merged 2024-04-26T20:32:42.235 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-04-26T20:32:42.235 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:32:42.235 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:32:42.235 INFO:teuthology.orchestra.run.smithi102.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-26T20:32:42.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:42.236 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-04-26T20:32:42.285 INFO:teuthology.orchestra.run.smithi102.stdout:114 2024-04-26T20:32:42.285 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-04-26T20:32:42.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo stdin-killer -- python3 -c ' 2024-04-26T20:32:42.286 DEBUG:teuthology.orchestra.run.smithi102:> import glob 2024-04-26T20:32:42.286 DEBUG:teuthology.orchestra.run.smithi102:> import re 2024-04-26T20:32:42.286 DEBUG:teuthology.orchestra.run.smithi102:> import os 2024-04-26T20:32:42.286 DEBUG:teuthology.orchestra.run.smithi102:> import subprocess 2024-04-26T20:32:42.286 DEBUG:teuthology.orchestra.run.smithi102:> 2024-04-26T20:32:42.286 DEBUG:teuthology.orchestra.run.smithi102:> def _find_admin_socket(client_name): 2024-04-26T20:32:42.286 DEBUG:teuthology.orchestra.run.smithi102:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-26T20:32:42.286 DEBUG:teuthology.orchestra.run.smithi102:> files = glob.glob(asok_path) 2024-04-26T20:32:42.286 DEBUG:teuthology.orchestra.run.smithi102:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> # Given a non-glob path, it better be there 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> if "*" not in asok_path: 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> assert(len(files) == 1) 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> return files[0] 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> for f in files: 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> contents = proc_f.read() 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> if mountpoint in contents: 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> return f 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> 2024-04-26T20:32:42.287 DEBUG:teuthology.orchestra.run.smithi102:> print(_find_admin_socket("client.admin")) 2024-04-26T20:32:42.288 DEBUG:teuthology.orchestra.run.smithi102:> ' 2024-04-26T20:32:42.439 INFO:teuthology.orchestra.run.smithi102.stdout:/var/run/ceph/ceph-client.admin.71624.asok 2024-04-26T20:32:42.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:32:42.455 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.71624.asok 2024-04-26T20:32:42.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:42.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.71624.asok status 2024-04-26T20:32:42.677 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:32:42.678 INFO:teuthology.orchestra.run.smithi102.stdout: "metadata": { 2024-04-26T20:32:42.678 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_sha1": "bc612ffb9c1d5bc27536bc917a705262a36d4387", 2024-04-26T20:32:42.678 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T20:32:42.678 INFO:teuthology.orchestra.run.smithi102.stdout: "entity_id": "admin", 2024-04-26T20:32:42.678 INFO:teuthology.orchestra.run.smithi102.stdout: "hostname": "smithi102", 2024-04-26T20:32:42.678 INFO:teuthology.orchestra.run.smithi102.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-26T20:32:42.678 INFO:teuthology.orchestra.run.smithi102.stdout: "pid": "71624", 2024-04-26T20:32:42.678 INFO:teuthology.orchestra.run.smithi102.stdout: "root": "/" 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "dentry_count": 0, 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "dentry_pinned_count": 0, 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24719, 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "inst": { 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "name": { 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "client", 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 24719 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": { 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:0", 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 1441260705 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-26T20:32:42.679 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": { 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:0", 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 1441260705 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "inst_str": "client.24719 172.21.15.102:0/1441260705", 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "addr_str": "172.21.15.102:0/1441260705", 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "inode_count": 1, 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_epoch": 16, 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_epoch": 89, 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_epoch_barrier": 0, 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "blocklisted": false, 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs" 2024-04-26T20:32:42.680 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:32:42.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:42.692 DEBUG:teuthology.orchestra.run.smithi102:> stat --file-system '--printf=%T 2024-04-26T20:32:42.692 DEBUG:teuthology.orchestra.run.smithi102:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:32:42.743 INFO:teuthology.orchestra.run.smithi102.stdout:fuseblk 2024-04-26T20:32:42.743 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:32:42.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:42.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:32:42.822 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-26T20:32:42.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T20:32:42.966 INFO:teuthology.orchestra.run.smithi102.stdout:# file: . 2024-04-26T20:32:42.966 INFO:teuthology.orchestra.run.smithi102.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-26T20:32:42.966 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:42.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:32:42.976 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:32:42.976 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T20:32:43.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:32:43.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.227+0000 7f7c49400640 1 -- 172.21.15.102:0/620773085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c4410ab40 msgr2=0x7f7c441113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:43.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.227+0000 7f7c49400640 1 --2- 172.21.15.102:0/620773085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c4410ab40 0x7f7c441113d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009be0 tx=0x7f7c3c02a6c0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.227+0000 7f7c49400640 1 -- 172.21.15.102:0/620773085 shutdown_connections 2024-04-26T20:32:43.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.227+0000 7f7c49400640 1 --2- 172.21.15.102:0/620773085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c4410a180 0x7f7c4410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.227+0000 7f7c49400640 1 --2- 172.21.15.102:0/620773085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c4410ab40 0x7f7c441113d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.227+0000 7f7c49400640 1 --2- 172.21.15.102:0/620773085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c44108f90 0x7f7c441093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.227+0000 7f7c49400640 1 -- 172.21.15.102:0/620773085 >> 172.21.15.102:0/620773085 conn(0x7f7c44077f40 msgr2=0x7f7c4407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:43.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.227+0000 7f7c49400640 1 -- 172.21.15.102:0/620773085 shutdown_connections 2024-04-26T20:32:43.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.227+0000 7f7c49400640 1 -- 172.21.15.102:0/620773085 wait complete. 2024-04-26T20:32:43.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.228+0000 7f7c49400640 1 Processor -- start 2024-04-26T20:32:43.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.228+0000 7f7c49400640 1 -- start start 2024-04-26T20:32:43.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c49400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44108f90 0x7f7c4419f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c49400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c4410a180 0x7f7c4419f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c49400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c4410ab40 0x7f7c441a66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c49400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7c44114ad0 con 0x7f7c44108f90 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c49400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7c44114950 con 0x7f7c4410ab40 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c49400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7c44114c50 con 0x7f7c4410a180 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c42800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44108f90 0x7f7c4419f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c42800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44108f90 0x7f7c4419f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33984/0 (socket says 172.21.15.102:33984) 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c42800640 1 -- 172.21.15.102:0/2917735105 learned_addr learned my addr 172.21.15.102:0/2917735105 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c41e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c4410a180 0x7f7c4419f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.229+0000 7f7c43200640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c4410ab40 0x7f7c441a66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c42800640 1 -- 172.21.15.102:0/2917735105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c4410a180 msgr2=0x7f7c4419f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:43.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c42800640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c4410a180 0x7f7c4419f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c42800640 1 -- 172.21.15.102:0/2917735105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c4410ab40 msgr2=0x7f7c441a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c42800640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c4410ab40 0x7f7c441a66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c42800640 1 -- 172.21.15.102:0/2917735105 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c441a6d40 con 0x7f7c44108f90 2024-04-26T20:32:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c42800640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44108f90 0x7f7c4419f120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400e9c0 tx=0x7f7c3400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c3b400640 1 -- 172.21.15.102:0/2917735105 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3400cd80 con 0x7f7c44108f90 2024-04-26T20:32:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c3b400640 1 -- 172.21.15.102:0/2917735105 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c340035f0 con 0x7f7c44108f90 2024-04-26T20:32:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c49400640 1 -- 172.21.15.102:0/2917735105 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c441a6fd0 con 0x7f7c44108f90 2024-04-26T20:32:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c3b400640 1 -- 172.21.15.102:0/2917735105 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c34010690 con 0x7f7c44108f90 2024-04-26T20:32:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.230+0000 7f7c49400640 1 -- 172.21.15.102:0/2917735105 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c441a7440 con 0x7f7c44108f90 2024-04-26T20:32:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.232+0000 7f7c49400640 1 -- 172.21.15.102:0/2917735105 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c441102b0 con 0x7f7c44108f90 2024-04-26T20:32:43.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.233+0000 7f7c3b400640 1 -- 172.21.15.102:0/2917735105 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f7c34004000 con 0x7f7c44108f90 2024-04-26T20:32:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.235+0000 7f7c3b400640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7c280a3bb0 0x7f7c280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.235+0000 7f7c3b400640 1 -- 172.21.15.102:0/2917735105 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7c340d0b80 con 0x7f7c44108f90 2024-04-26T20:32:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.235+0000 7f7c41e00640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7c280a3bb0 0x7f7c280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:43.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.236+0000 7f7c41e00640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7c280a3bb0 0x7f7c280a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7c441a0630 tx=0x7f7c30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:43.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.240+0000 7f7c3b400640 1 -- 172.21.15.102:0/2917735105 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7c3409d2c0 con 0x7f7c44108f90 2024-04-26T20:32:43.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.445+0000 7f7c49400640 1 -- 172.21.15.102:0/2917735105 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c440635d0 con 0x7f7c44108f90 2024-04-26T20:32:43.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.448+0000 7f7c3b400640 1 -- 172.21.15.102:0/2917735105 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24164 (secure 0 0 0) 0x7f7c3409cbc0 con 0x7f7c44108f90 2024-04-26T20:32:43.450 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:43.450 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:32:30.827511+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:32:43.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.451+0000 7f7c49400640 1 -- 172.21.15.102:0/2917735105 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7c280a3bb0 msgr2=0x7f7c280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.451+0000 7f7c49400640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7c280a3bb0 0x7f7c280a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7c441a0630 tx=0x7f7c30009260 comp rx=0 tx=0).stop 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.451+0000 7f7c49400640 1 -- 172.21.15.102:0/2917735105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44108f90 msgr2=0x7f7c4419f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.451+0000 7f7c49400640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44108f90 0x7f7c4419f120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400e9c0 tx=0x7f7c3400ee80 comp rx=0 tx=0).stop 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.451+0000 7f7c49400640 1 -- 172.21.15.102:0/2917735105 shutdown_connections 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.451+0000 7f7c49400640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7c280a3bb0 0x7f7c280a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.451+0000 7f7c49400640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c4410ab40 0x7f7c441a66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.451+0000 7f7c49400640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c4410a180 0x7f7c4419f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.451+0000 7f7c49400640 1 --2- 172.21.15.102:0/2917735105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44108f90 0x7f7c4419f120 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.451+0000 7f7c49400640 1 -- 172.21.15.102:0/2917735105 >> 172.21.15.102:0/2917735105 conn(0x7f7c44077f40 msgr2=0x7f7c4410d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.452+0000 7f7c49400640 1 -- 172.21.15.102:0/2917735105 shutdown_connections 2024-04-26T20:32:43.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.452+0000 7f7c49400640 1 -- 172.21.15.102:0/2917735105 wait complete. 2024-04-26T20:32:43.473 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:27:10.527933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:32:23.439237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:32:24.702263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:32:28.348785+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:32:43.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:32:43.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.672+0000 7f8fff200640 1 -- 172.21.15.102:0/3483331350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900010ee10 msgr2=0x7f9000111200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:43.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.672+0000 7f8fff200640 1 --2- 172.21.15.102:0/3483331350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900010ee10 0x7f9000111200 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009cd0 tx=0x7f8ff802a8e0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.672+0000 7f8fff200640 1 -- 172.21.15.102:0/3483331350 shutdown_connections 2024-04-26T20:32:43.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.672+0000 7f8fff200640 1 --2- 172.21.15.102:0/3483331350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900010ee10 0x7f9000111200 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.672+0000 7f8fff200640 1 --2- 172.21.15.102:0/3483331350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9000075390 0x7f9000075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.672+0000 7f8fff200640 1 --2- 172.21.15.102:0/3483331350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9000076970 0x7f9000074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.672+0000 7f8fff200640 1 -- 172.21.15.102:0/3483331350 >> 172.21.15.102:0/3483331350 conn(0x7f90000fddd0 msgr2=0x7f90001001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:43.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.672+0000 7f8fff200640 1 -- 172.21.15.102:0/3483331350 shutdown_connections 2024-04-26T20:32:43.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.672+0000 7f8fff200640 1 -- 172.21.15.102:0/3483331350 wait complete. 2024-04-26T20:32:43.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.673+0000 7f8fff200640 1 Processor -- start 2024-04-26T20:32:43.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.673+0000 7f8fff200640 1 -- start start 2024-04-26T20:32:43.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.673+0000 7f8fff200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9000075390 0x7f900019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:43.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.673+0000 7f8fff200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9000076970 0x7f900019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:43.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.673+0000 7f8fff200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900010ee10 0x7f90001a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:43.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.673+0000 7f8fff200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f90001149b0 con 0x7f900010ee10 2024-04-26T20:32:43.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.673+0000 7f8fff200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9000114830 con 0x7f9000075390 2024-04-26T20:32:43.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.673+0000 7f8fff200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9000114b30 con 0x7f9000076970 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffde00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9000075390 0x7f900019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffde00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9000075390 0x7f900019f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:57008/0 (socket says 172.21.15.102:57008) 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffde00640 1 -- 172.21.15.102:0/3783853502 learned_addr learned my addr 172.21.15.102:0/3783853502 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffd400640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9000076970 0x7f900019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffe800640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900010ee10 0x7f90001a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffde00640 1 -- 172.21.15.102:0/3783853502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9000076970 msgr2=0x7f900019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffde00640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9000076970 0x7f900019f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffde00640 1 -- 172.21.15.102:0/3783853502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900010ee10 msgr2=0x7f90001a65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffde00640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900010ee10 0x7f90001a65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffde00640 1 -- 172.21.15.102:0/3783853502 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90001a6c20 con 0x7f9000075390 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.674+0000 7f8ffde00640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9000075390 0x7f900019f000 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000e9f0 tx=0x7f8ff000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.675+0000 7f8ff6a00640 1 -- 172.21.15.102:0/3783853502 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff000cd80 con 0x7f9000075390 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.675+0000 7f8ff6a00640 1 -- 172.21.15.102:0/3783853502 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ff00054d0 con 0x7f9000075390 2024-04-26T20:32:43.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.675+0000 7f8fff200640 1 -- 172.21.15.102:0/3783853502 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90001a6f10 con 0x7f9000075390 2024-04-26T20:32:43.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.675+0000 7f8ff6a00640 1 -- 172.21.15.102:0/3783853502 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0010690 con 0x7f9000075390 2024-04-26T20:32:43.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.675+0000 7f8fff200640 1 -- 172.21.15.102:0/3783853502 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90001a7370 con 0x7f9000075390 2024-04-26T20:32:43.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.676+0000 7f8fff200640 1 -- 172.21.15.102:0/3783853502 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9000069af0 con 0x7f9000075390 2024-04-26T20:32:43.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.678+0000 7f8ff6a00640 1 -- 172.21.15.102:0/3783853502 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f8ff0004e90 con 0x7f9000075390 2024-04-26T20:32:43.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.679+0000 7f8ff6a00640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f8fe40a3c30 0x7f8fe40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:43.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.679+0000 7f8ff6a00640 1 -- 172.21.15.102:0/3783853502 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8ff00d0a30 con 0x7f9000075390 2024-04-26T20:32:43.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.679+0000 7f8ffd400640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f8fe40a3c30 0x7f8fe40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:43.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.680+0000 7f8ffd400640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f8fe40a3c30 0x7f8fe40a60f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0095f0 tx=0x7f8fec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:43.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.683+0000 7f8ff6a00640 1 -- 172.21.15.102:0/3783853502 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9000069af0 con 0x7f9000075390 2024-04-26T20:32:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:43 smithi130 ceph-mon[32704]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:43 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2917735105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:43.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.883+0000 7f8fff200640 1 -- 172.21.15.102:0/3783853502 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f90000635d0 con 0x7f9000075390 2024-04-26T20:32:43.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.884+0000 7f8ff6a00640 1 -- 172.21.15.102:0/3783853502 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f8ff009d170 con 0x7f9000075390 2024-04-26T20:32:43.885 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-26T20:32:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 -- 172.21.15.102:0/3783853502 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f8fe40a3c30 msgr2=0x7f8fe40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f8fe40a3c30 0x7f8fe40a60f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0095f0 tx=0x7f8fec005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 -- 172.21.15.102:0/3783853502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9000075390 msgr2=0x7f900019f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9000075390 0x7f900019f000 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000e9f0 tx=0x7f8ff000edf0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 -- 172.21.15.102:0/3783853502 shutdown_connections 2024-04-26T20:32:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f8fe40a3c30 0x7f8fe40a60f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900010ee10 0x7f90001a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9000076970 0x7f900019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 --2- 172.21.15.102:0/3783853502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9000075390 0x7f900019f000 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 -- 172.21.15.102:0/3783853502 >> 172.21.15.102:0/3783853502 conn(0x7f90000fddd0 msgr2=0x7f90001020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 -- 172.21.15.102:0/3783853502 shutdown_connections 2024-04-26T20:32:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:43.887+0000 7f8fff200640 1 -- 172.21.15.102:0/3783853502 wait complete. 2024-04-26T20:32:43.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:43 smithi102 ceph-mon[28094]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:43.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:43 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2917735105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:43.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:32:44.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:43 smithi112 ceph-mon[32769]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:32:44.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:43 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2917735105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:44.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.116+0000 7fee56c00640 1 -- 172.21.15.102:0/533785331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee50075310 msgr2=0x7fee50075770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:44.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.116+0000 7fee56c00640 1 --2- 172.21.15.102:0/533785331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee50075310 0x7fee50075770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009900 tx=0x7fee3c02a480 comp rx=0 tx=0).stop 2024-04-26T20:32:44.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.116+0000 7fee56c00640 1 -- 172.21.15.102:0/533785331 shutdown_connections 2024-04-26T20:32:44.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.116+0000 7fee56c00640 1 --2- 172.21.15.102:0/533785331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee5010a6f0 0x7fee5010cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.116+0000 7fee56c00640 1 --2- 172.21.15.102:0/533785331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee50075310 0x7fee50075770 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.116+0000 7fee56c00640 1 --2- 172.21.15.102:0/533785331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee500768f0 0x7fee50074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.116+0000 7fee56c00640 1 -- 172.21.15.102:0/533785331 >> 172.21.15.102:0/533785331 conn(0x7fee500fddd0 msgr2=0x7fee501001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:44.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.116+0000 7fee56c00640 1 -- 172.21.15.102:0/533785331 shutdown_connections 2024-04-26T20:32:44.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.116+0000 7fee56c00640 1 -- 172.21.15.102:0/533785331 wait complete. 2024-04-26T20:32:44.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.117+0000 7fee56c00640 1 Processor -- start 2024-04-26T20:32:44.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee56c00640 1 -- start start 2024-04-26T20:32:44.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee56c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee50075310 0x7fee5019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:44.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee56c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee500768f0 0x7fee5019b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:44.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee56c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee5010a6f0 0x7fee501a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee56c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fee50110580 con 0x7fee500768f0 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee56c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fee50110400 con 0x7fee5010a6f0 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee56c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fee50110700 con 0x7fee50075310 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee55800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee50075310 0x7fee5019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee54e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee500768f0 0x7fee5019b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee55800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee50075310 0x7fee5019ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:58508/0 (socket says 172.21.15.102:58508) 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee55800640 1 -- 172.21.15.102:0/1364174164 learned_addr learned my addr 172.21.15.102:0/1364174164 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.118+0000 7fee56200640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee5010a6f0 0x7fee501a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee55800640 1 -- 172.21.15.102:0/1364174164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee5010a6f0 msgr2=0x7fee501a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee55800640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee5010a6f0 0x7fee501a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee55800640 1 -- 172.21.15.102:0/1364174164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee500768f0 msgr2=0x7fee5019b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee55800640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee500768f0 0x7fee5019b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee55800640 1 -- 172.21.15.102:0/1364174164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee501a2870 con 0x7fee50075310 2024-04-26T20:32:44.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee54e00640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee500768f0 0x7fee5019b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:44.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee55800640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee50075310 0x7fee5019ac60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee4400e9f0 tx=0x7fee4400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:44.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee4e000640 1 -- 172.21.15.102:0/1364174164 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4400cd80 con 0x7fee50075310 2024-04-26T20:32:44.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee4e000640 1 -- 172.21.15.102:0/1364174164 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee440108e0 con 0x7fee50075310 2024-04-26T20:32:44.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee56c00640 1 -- 172.21.15.102:0/1364174164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee501a2b60 con 0x7fee50075310 2024-04-26T20:32:44.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee4e000640 1 -- 172.21.15.102:0/1364174164 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee44018980 con 0x7fee50075310 2024-04-26T20:32:44.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.119+0000 7fee56c00640 1 -- 172.21.15.102:0/1364174164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee501a30b0 con 0x7fee50075310 2024-04-26T20:32:44.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.120+0000 7fee56c00640 1 -- 172.21.15.102:0/1364174164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee501094b0 con 0x7fee50075310 2024-04-26T20:32:44.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.122+0000 7fee4e000640 1 -- 172.21.15.102:0/1364174164 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fee44010430 con 0x7fee50075310 2024-04-26T20:32:44.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.124+0000 7fee4e000640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3bb0 0x7fee380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:44.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.124+0000 7fee4e000640 1 -- 172.21.15.102:0/1364174164 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fee440d0a90 con 0x7fee50075310 2024-04-26T20:32:44.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.124+0000 7fee54e00640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3bb0 0x7fee380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:44.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.125+0000 7fee54e00640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3bb0 0x7fee380a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000c00 tx=0x7fee3c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.132+0000 7fee4e000640 1 -- 172.21.15.102:0/1364174164 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fee4409d1d0 con 0x7fee50075310 2024-04-26T20:32:44.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.333+0000 7fee56c00640 1 -- 172.21.15.102:0/1364174164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fee50063530 con 0x7fee50075310 2024-04-26T20:32:44.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.334+0000 7fee4e000640 1 -- 172.21.15.102:0/1364174164 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fee4409cad0 con 0x7fee50075310 2024-04-26T20:32:44.335 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:32:44.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 -- 172.21.15.102:0/1364174164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3bb0 msgr2=0x7fee380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:44.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3bb0 0x7fee380a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000c00 tx=0x7fee3c0314a0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 -- 172.21.15.102:0/1364174164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee50075310 msgr2=0x7fee5019ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee50075310 0x7fee5019ac60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee4400e9f0 tx=0x7fee4400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 -- 172.21.15.102:0/1364174164 shutdown_connections 2024-04-26T20:32:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fee380a3bb0 0x7fee380a6070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee500768f0 0x7fee5019b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee5010a6f0 0x7fee501a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 --2- 172.21.15.102:0/1364174164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee50075310 0x7fee5019ac60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 -- 172.21.15.102:0/1364174164 >> 172.21.15.102:0/1364174164 conn(0x7fee500fddd0 msgr2=0x7fee50106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 -- 172.21.15.102:0/1364174164 shutdown_connections 2024-04-26T20:32:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.337+0000 7fee56c00640 1 -- 172.21.15.102:0/1364174164 wait complete. 2024-04-26T20:32:44.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:32:44.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.564+0000 7f51f0c00640 1 -- 172.21.15.102:0/795515789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51ec102960 msgr2=0x7f51ec102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:44.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.564+0000 7f51f0c00640 1 --2- 172.21.15.102:0/795515789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51ec102960 0x7f51ec102d80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009900 tx=0x7f51dc02a400 comp rx=0 tx=0).stop 2024-04-26T20:32:44.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.564+0000 7f51f0c00640 1 -- 172.21.15.102:0/795515789 shutdown_connections 2024-04-26T20:32:44.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.564+0000 7f51f0c00640 1 --2- 172.21.15.102:0/795515789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec104510 0x7f51ec10ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.564+0000 7f51f0c00640 1 --2- 172.21.15.102:0/795515789 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51ec103b50 0x7f51ec103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.564+0000 7f51f0c00640 1 --2- 172.21.15.102:0/795515789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51ec102960 0x7f51ec102d80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.564+0000 7f51f0c00640 1 -- 172.21.15.102:0/795515789 >> 172.21.15.102:0/795515789 conn(0x7f51ec0fdef0 msgr2=0x7f51ec100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:44.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.564+0000 7f51f0c00640 1 -- 172.21.15.102:0/795515789 shutdown_connections 2024-04-26T20:32:44.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.564+0000 7f51f0c00640 1 -- 172.21.15.102:0/795515789 wait complete. 2024-04-26T20:32:44.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.565+0000 7f51f0c00640 1 Processor -- start 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51f0c00640 1 -- start start 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51f0c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51ec102960 0x7f51ec0766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51f0c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51ec103b50 0x7f51ec074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51f0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec104510 0x7f51ec075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51f0c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec10e460 con 0x7f51ec104510 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51f0c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec10e2e0 con 0x7f51ec103b50 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51f0c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec10e5e0 con 0x7f51ec102960 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51e9e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51ec102960 0x7f51ec0766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51e9e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51ec102960 0x7f51ec0766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:58532/0 (socket says 172.21.15.102:58532) 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51e9e00640 1 -- 172.21.15.102:0/3751487568 learned_addr learned my addr 172.21.15.102:0/3751487568 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51e1400640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51ec103b50 0x7f51ec074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:44.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.566+0000 7f51ea800640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec104510 0x7f51ec075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51e9e00640 1 -- 172.21.15.102:0/3751487568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51ec103b50 msgr2=0x7f51ec074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51e9e00640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51ec103b50 0x7f51ec074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51e9e00640 1 -- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec104510 msgr2=0x7f51ec075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51e9e00640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec104510 0x7f51ec075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51e9e00640 1 -- 172.21.15.102:0/3751487568 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec075af0 con 0x7f51ec102960 2024-04-26T20:32:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51e1400640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51ec103b50 0x7f51ec074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51ea800640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec104510 0x7f51ec075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:44.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51e9e00640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51ec102960 0x7f51ec0766e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f51dc006010 tx=0x7f51dc0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:44.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51e3400640 1 -- 172.21.15.102:0/3751487568 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc03f030 con 0x7f51ec102960 2024-04-26T20:32:44.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51f0c00640 1 -- 172.21.15.102:0/3751487568 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ec1a2c20 con 0x7f51ec102960 2024-04-26T20:32:44.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51e3400640 1 -- 172.21.15.102:0/3751487568 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51dc031910 con 0x7f51ec102960 2024-04-26T20:32:44.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51e3400640 1 -- 172.21.15.102:0/3751487568 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc035070 con 0x7f51ec102960 2024-04-26T20:32:44.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.567+0000 7f51f0c00640 1 -- 172.21.15.102:0/3751487568 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ec1a3170 con 0x7f51ec102960 2024-04-26T20:32:44.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.569+0000 7f51f0c00640 1 -- 172.21.15.102:0/3751487568 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51ec109d20 con 0x7f51ec102960 2024-04-26T20:32:44.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.571+0000 7f51e3400640 1 -- 172.21.15.102:0/3751487568 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f51dc042030 con 0x7f51ec102960 2024-04-26T20:32:44.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.572+0000 7f51e3400640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f51d00a3bb0 0x7f51d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:44.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.572+0000 7f51e3400640 1 -- 172.21.15.102:0/3751487568 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f51dc0f0e50 con 0x7f51ec102960 2024-04-26T20:32:44.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.572+0000 7f51e1400640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f51d00a3bb0 0x7f51d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:44.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.573+0000 7f51e1400640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f51d00a3bb0 0x7f51d00a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f51ec076450 tx=0x7f51d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:44.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.579+0000 7f51e3400640 1 -- 172.21.15.102:0/3751487568 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f51dc0bd5d0 con 0x7f51ec102960 2024-04-26T20:32:44.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:44 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3783853502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:32:44.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:44 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1364174164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:32:44.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.785+0000 7f51f0c00640 1 -- 172.21.15.102:0/3751487568 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f51ec02dd20 con 0x7f51ec102960 2024-04-26T20:32:44.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.786+0000 7f51e3400640 1 -- 172.21.15.102:0/3751487568 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f51dc0bced0 con 0x7f51ec102960 2024-04-26T20:32:44.787 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-26T20:32:44.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.788+0000 7f51f0c00640 1 -- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f51d00a3bb0 msgr2=0x7f51d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:44.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.788+0000 7f51f0c00640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f51d00a3bb0 0x7f51d00a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f51ec076450 tx=0x7f51d4005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:44.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.789+0000 7f51f0c00640 1 -- 172.21.15.102:0/3751487568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51ec102960 msgr2=0x7f51ec0766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:44.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.789+0000 7f51f0c00640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51ec102960 0x7f51ec0766e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f51dc006010 tx=0x7f51dc0042f0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.789+0000 7f51f0c00640 1 -- 172.21.15.102:0/3751487568 shutdown_connections 2024-04-26T20:32:44.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.789+0000 7f51f0c00640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f51d00a3bb0 0x7f51d00a6070 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.789+0000 7f51f0c00640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec104510 0x7f51ec075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.789+0000 7f51f0c00640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51ec103b50 0x7f51ec074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.789+0000 7f51f0c00640 1 --2- 172.21.15.102:0/3751487568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51ec102960 0x7f51ec0766e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:44.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.789+0000 7f51f0c00640 1 -- 172.21.15.102:0/3751487568 >> 172.21.15.102:0/3751487568 conn(0x7f51ec0fdef0 msgr2=0x7f51ec106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:44.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.789+0000 7f51f0c00640 1 -- 172.21.15.102:0/3751487568 shutdown_connections 2024-04-26T20:32:44.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:44.789+0000 7f51f0c00640 1 -- 172.21.15.102:0/3751487568 wait complete. 2024-04-26T20:32:44.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:32:44.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:44 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3783853502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:32:44.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:44 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1364174164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:32:45.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.014+0000 7f353a200640 1 -- 172.21.15.102:0/1244676131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f353410ab40 msgr2=0x7f35341113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:45.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.014+0000 7f353a200640 1 --2- 172.21.15.102:0/1244676131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f353410ab40 0x7f35341113d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f352c009be0 tx=0x7f352c02a6c0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.014+0000 7f353a200640 1 -- 172.21.15.102:0/1244676131 shutdown_connections 2024-04-26T20:32:45.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.014+0000 7f353a200640 1 --2- 172.21.15.102:0/1244676131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f353410ab40 0x7f35341113d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.014+0000 7f353a200640 1 --2- 172.21.15.102:0/1244676131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f353410a180 0x7f353410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.014+0000 7f353a200640 1 --2- 172.21.15.102:0/1244676131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3534108f90 0x7f35341093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.014+0000 7f353a200640 1 -- 172.21.15.102:0/1244676131 >> 172.21.15.102:0/1244676131 conn(0x7f3534077f30 msgr2=0x7f353407a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:45.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.014+0000 7f353a200640 1 -- 172.21.15.102:0/1244676131 shutdown_connections 2024-04-26T20:32:45.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.014+0000 7f353a200640 1 -- 172.21.15.102:0/1244676131 wait complete. 2024-04-26T20:32:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.015+0000 7f353a200640 1 Processor -- start 2024-04-26T20:32:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.015+0000 7f353a200640 1 -- start start 2024-04-26T20:32:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.015+0000 7f353a200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534108f90 0x7f353419f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.015+0000 7f353a200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f353410a180 0x7f353419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.015+0000 7f353a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353410ab40 0x7f35341a6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f353a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3534114a90 con 0x7f353410ab40 2024-04-26T20:32:45.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f353a200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3534114910 con 0x7f3534108f90 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f353a200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3534114c10 con 0x7f353410a180 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f3533400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534108f90 0x7f353419f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f352be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f353410a180 0x7f353419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f3533400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534108f90 0x7f353419f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:37154/0 (socket says 172.21.15.102:37154) 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f3533e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353410ab40 0x7f35341a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f3533400640 1 -- 172.21.15.102:0/1034038601 learned_addr learned my addr 172.21.15.102:0/1034038601 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f352be00640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f353410a180 0x7f353419f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:36670/0 (socket says 172.21.15.102:36670) 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f3533400640 1 -- 172.21.15.102:0/1034038601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f353410a180 msgr2=0x7f353419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f3533400640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f353410a180 0x7f353419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f3533400640 1 -- 172.21.15.102:0/1034038601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353410ab40 msgr2=0x7f35341a6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f3533400640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353410ab40 0x7f35341a6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.016+0000 7f3533400640 1 -- 172.21.15.102:0/1034038601 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35341a6d80 con 0x7f3534108f90 2024-04-26T20:32:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.017+0000 7f3533400640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534108f90 0x7f353419f160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f352400e9f0 tx=0x7f352400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.017+0000 7f3530c00640 1 -- 172.21.15.102:0/1034038601 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352400ccc0 con 0x7f3534108f90 2024-04-26T20:32:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.017+0000 7f3530c00640 1 -- 172.21.15.102:0/1034038601 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35240035f0 con 0x7f3534108f90 2024-04-26T20:32:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.017+0000 7f353a200640 1 -- 172.21.15.102:0/1034038601 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35341a7070 con 0x7f3534108f90 2024-04-26T20:32:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.017+0000 7f3530c00640 1 -- 172.21.15.102:0/1034038601 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352400ce60 con 0x7f3534108f90 2024-04-26T20:32:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.017+0000 7f353a200640 1 -- 172.21.15.102:0/1034038601 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35341a75c0 con 0x7f3534108f90 2024-04-26T20:32:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.018+0000 7f353a200640 1 -- 172.21.15.102:0/1034038601 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3534110380 con 0x7f3534108f90 2024-04-26T20:32:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.020+0000 7f3530c00640 1 -- 172.21.15.102:0/1034038601 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f3524004000 con 0x7f3534108f90 2024-04-26T20:32:45.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.021+0000 7f3530c00640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f35180a3bb0 0x7f35180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.022+0000 7f3530c00640 1 -- 172.21.15.102:0/1034038601 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f35240d0880 con 0x7f3534108f90 2024-04-26T20:32:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.022+0000 7f352be00640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f35180a3bb0 0x7f35180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.022+0000 7f352be00640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f35180a3bb0 0x7f35180a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f35341a0670 tx=0x7f351c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:45.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.025+0000 7f3530c00640 1 -- 172.21.15.102:0/1034038601 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f352409cfc0 con 0x7f3534108f90 2024-04-26T20:32:45.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:44 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3783853502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:32:45.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:44 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1364174164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:32:45.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.227+0000 7f353a200640 1 -- 172.21.15.102:0/1034038601 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3534063560 con 0x7f3534108f90 2024-04-26T20:32:45.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.228+0000 7f3530c00640 1 -- 172.21.15.102:0/1034038601 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f352409c8c0 con 0x7f3534108f90 2024-04-26T20:32:45.229 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:32:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 -- 172.21.15.102:0/1034038601 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f35180a3bb0 msgr2=0x7f35180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f35180a3bb0 0x7f35180a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f35341a0670 tx=0x7f351c005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:45.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 -- 172.21.15.102:0/1034038601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534108f90 msgr2=0x7f353419f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:45.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534108f90 0x7f353419f160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f352400e9f0 tx=0x7f352400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 -- 172.21.15.102:0/1034038601 shutdown_connections 2024-04-26T20:32:45.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f35180a3bb0 0x7f35180a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353410ab40 0x7f35341a6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f353410a180 0x7f353419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 --2- 172.21.15.102:0/1034038601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534108f90 0x7f353419f160 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 -- 172.21.15.102:0/1034038601 >> 172.21.15.102:0/1034038601 conn(0x7f3534077f30 msgr2=0x7f353410d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:45.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 -- 172.21.15.102:0/1034038601 shutdown_connections 2024-04-26T20:32:45.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.231+0000 7f353a200640 1 -- 172.21.15.102:0/1034038601 wait complete. 2024-04-26T20:32:45.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T20:32:45.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.456+0000 7f34b0c00640 1 -- 172.21.15.102:0/1385233398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34ac078ed0 msgr2=0x7f34ac079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:45.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.456+0000 7f34b0c00640 1 --2- 172.21.15.102:0/1385233398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34ac078ed0 0x7f34ac079770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f34a4009ca0 tx=0x7f34a402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.456+0000 7f34b0c00640 1 -- 172.21.15.102:0/1385233398 shutdown_connections 2024-04-26T20:32:45.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.456+0000 7f34b0c00640 1 --2- 172.21.15.102:0/1385233398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34ac078ed0 0x7f34ac079770 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.456+0000 7f34b0c00640 1 --2- 172.21.15.102:0/1385233398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34ac078530 0x7f34ac078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.456+0000 7f34b0c00640 1 --2- 172.21.15.102:0/1385233398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34ac0772f0 0x7f34ac077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.456+0000 7f34b0c00640 1 -- 172.21.15.102:0/1385233398 >> 172.21.15.102:0/1385233398 conn(0x7f34ac1000f0 msgr2=0x7f34ac102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:45.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.456+0000 7f34b0c00640 1 -- 172.21.15.102:0/1385233398 shutdown_connections 2024-04-26T20:32:45.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.456+0000 7f34b0c00640 1 -- 172.21.15.102:0/1385233398 wait complete. 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.457+0000 7f34b0c00640 1 Processor -- start 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.457+0000 7f34b0c00640 1 -- start start 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.457+0000 7f34b0c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34ac0772f0 0x7f34ac19f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.457+0000 7f34b0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34ac078530 0x7f34ac19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.457+0000 7f34b0c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34ac078ed0 0x7f34ac1a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.457+0000 7f34b0c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac114ac0 con 0x7f34ac078530 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.457+0000 7f34b0c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac114940 con 0x7f34ac0772f0 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.457+0000 7f34b0c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac114c40 con 0x7f34ac078ed0 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34aa800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34ac078ed0 0x7f34ac1a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34a9e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34ac0772f0 0x7f34ac19f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:45.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34a9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34ac078530 0x7f34ac19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34aa800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34ac078ed0 0x7f34ac1a6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:36694/0 (socket says 172.21.15.102:36694) 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34aa800640 1 -- 172.21.15.102:0/3124712133 learned_addr learned my addr 172.21.15.102:0/3124712133 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34a9e00640 1 -- 172.21.15.102:0/3124712133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34ac078ed0 msgr2=0x7f34ac1a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34a9e00640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34ac078ed0 0x7f34ac1a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34a9e00640 1 -- 172.21.15.102:0/3124712133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34ac078530 msgr2=0x7f34ac19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34a9e00640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34ac078530 0x7f34ac19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34a9e00640 1 -- 172.21.15.102:0/3124712133 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ac1a6de0 con 0x7f34ac0772f0 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.458+0000 7f34a9e00640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34ac0772f0 0x7f34ac19f1c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f349c00e9f0 tx=0x7f349c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.459+0000 7f34a2a00640 1 -- 172.21.15.102:0/3124712133 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c00cd80 con 0x7f34ac0772f0 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.459+0000 7f34a2a00640 1 -- 172.21.15.102:0/3124712133 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f349c0035f0 con 0x7f34ac0772f0 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.459+0000 7f34a2a00640 1 -- 172.21.15.102:0/3124712133 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c010670 con 0x7f34ac0772f0 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.459+0000 7f34b0c00640 1 -- 172.21.15.102:0/3124712133 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ac1a70d0 con 0x7f34ac0772f0 2024-04-26T20:32:45.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.459+0000 7f34b0c00640 1 -- 172.21.15.102:0/3124712133 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34ac1a75d0 con 0x7f34ac0772f0 2024-04-26T20:32:45.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.460+0000 7f34b0c00640 1 -- 172.21.15.102:0/3124712133 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34ac1114f0 con 0x7f34ac0772f0 2024-04-26T20:32:45.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.462+0000 7f34a2a00640 1 -- 172.21.15.102:0/3124712133 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f349c0037a0 con 0x7f34ac0772f0 2024-04-26T20:32:45.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.463+0000 7f34a2a00640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f34900a3c30 0x7f34900a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:45.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.464+0000 7f34a2a00640 1 -- 172.21.15.102:0/3124712133 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f349c0d0870 con 0x7f34ac0772f0 2024-04-26T20:32:45.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.464+0000 7f34a9400640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f34900a3c30 0x7f34900a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:45.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.464+0000 7f34a9400640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f34900a3c30 0x7f34900a60f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f34980095f0 tx=0x7f3498005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:45.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.468+0000 7f34a2a00640 1 -- 172.21.15.102:0/3124712133 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f349c09cfb0 con 0x7f34ac0772f0 2024-04-26T20:32:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.684+0000 7f34b0c00640 1 -- 172.21.15.102:0/3124712133 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f34ac02dad0 con 0x7f34ac0772f0 2024-04-26T20:32:45.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.685+0000 7f34a2a00640 1 -- 172.21.15.102:0/3124712133 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+602 (secure 0 0 0) 0x7f349c09c8b0 con 0x7f34ac0772f0 2024-04-26T20:32:45.686 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2611652511 2024-04-27T20:24:28.376337+0000 2024-04-26T20:32:45.686 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/1441707916 2024-04-27T20:24:28.376337+0000 2024-04-26T20:32:45.686 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/1441707916 2024-04-27T20:24:28.376337+0000 2024-04-26T20:32:45.687 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2809626135 2024-04-27T20:24:28.376337+0000 2024-04-26T20:32:45.687 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2372720525 2024-04-27T20:23:53.695696+0000 2024-04-26T20:32:45.687 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2628958579 2024-04-27T20:23:53.695696+0000 2024-04-26T20:32:45.687 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/3976810658 2024-04-27T20:23:53.695696+0000 2024-04-26T20:32:45.687 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/3976810658 2024-04-27T20:23:53.695696+0000 2024-04-26T20:32:45.687 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/3271239188 2024-04-27T20:24:28.376337+0000 2024-04-26T20:32:45.687 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1384283564 2024-04-27T20:23:53.695696+0000 2024-04-26T20:32:45.687 INFO:teuthology.orchestra.run.smithi102.stderr:listed 10 entries 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.688+0000 7f34b0c00640 1 -- 172.21.15.102:0/3124712133 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f34900a3c30 msgr2=0x7f34900a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.688+0000 7f34b0c00640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f34900a3c30 0x7f34900a60f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f34980095f0 tx=0x7f3498005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.688+0000 7f34b0c00640 1 -- 172.21.15.102:0/3124712133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34ac0772f0 msgr2=0x7f34ac19f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.688+0000 7f34b0c00640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34ac0772f0 0x7f34ac19f1c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f349c00e9f0 tx=0x7f349c00eeb0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.688+0000 7f34b0c00640 1 -- 172.21.15.102:0/3124712133 shutdown_connections 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.688+0000 7f34b0c00640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f34900a3c30 0x7f34900a60f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.688+0000 7f34b0c00640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34ac078530 0x7f34ac19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.688+0000 7f34b0c00640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34ac078ed0 0x7f34ac1a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.688+0000 7f34b0c00640 1 --2- 172.21.15.102:0/3124712133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34ac0772f0 0x7f34ac19f1c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.688+0000 7f34b0c00640 1 -- 172.21.15.102:0/3124712133 >> 172.21.15.102:0/3124712133 conn(0x7f34ac1000f0 msgr2=0x7f34ac101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.689+0000 7f34b0c00640 1 -- 172.21.15.102:0/3124712133 shutdown_connections 2024-04-26T20:32:45.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:45.689+0000 7f34b0c00640 1 -- 172.21.15.102:0/3124712133 wait complete. 2024-04-26T20:32:45.709 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:32:45.709 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T20:32:45.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:32:45.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:45 smithi130 ceph-mon[32704]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T20:32:45.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:45 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3751487568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:32:45.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:45 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1034038601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:32:45.873 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:45 smithi102 ceph-mon[28094]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T20:32:45.873 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:45 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3751487568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:32:45.874 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:45 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1034038601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:32:46.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.006+0000 7f56fde00640 1 -- 172.21.15.102:0/293277469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56f8108d90 msgr2=0x7f56f81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.006+0000 7f56fde00640 1 --2- 172.21.15.102:0/293277469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56f8108d90 0x7f56f81091b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009860 tx=0x7f56ec02a340 comp rx=0 tx=0).stop 2024-04-26T20:32:46.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.006+0000 7f56fde00640 1 -- 172.21.15.102:0/293277469 shutdown_connections 2024-04-26T20:32:46.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.006+0000 7f56fde00640 1 --2- 172.21.15.102:0/293277469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56f810a940 0x7f56f81111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.006+0000 7f56fde00640 1 --2- 172.21.15.102:0/293277469 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56f8109f80 0x7f56f810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.006+0000 7f56fde00640 1 --2- 172.21.15.102:0/293277469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56f8108d90 0x7f56f81091b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.006+0000 7f56fde00640 1 -- 172.21.15.102:0/293277469 >> 172.21.15.102:0/293277469 conn(0x7f56f8077d30 msgr2=0x7f56f807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:46.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.006+0000 7f56fde00640 1 -- 172.21.15.102:0/293277469 shutdown_connections 2024-04-26T20:32:46.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.006+0000 7f56fde00640 1 -- 172.21.15.102:0/293277469 wait complete. 2024-04-26T20:32:46.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.007+0000 7f56fde00640 1 Processor -- start 2024-04-26T20:32:46.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56fde00640 1 -- start start 2024-04-26T20:32:46.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56fde00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56f8108d90 0x7f56f819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56fde00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56f8109f80 0x7f56f819f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56fde00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56f810a940 0x7f56f81a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56fde00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f56f81148b0 con 0x7f56f8108d90 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56fde00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f56f8114730 con 0x7f56f810a940 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56fde00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f56f8114a30 con 0x7f56f8109f80 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56fca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56f8108d90 0x7f56f819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56f7e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56f8109f80 0x7f56f819f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56f7e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56f8109f80 0x7f56f819f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:36716/0 (socket says 172.21.15.102:36716) 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56fd400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56f810a940 0x7f56f81a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.008+0000 7f56f7e00640 1 -- 172.21.15.102:0/1504706469 learned_addr learned my addr 172.21.15.102:0/1504706469 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56f7e00640 1 -- 172.21.15.102:0/1504706469 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56f810a940 msgr2=0x7f56f81a65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56f7e00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56f810a940 0x7f56f81a65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56f7e00640 1 -- 172.21.15.102:0/1504706469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56f8108d90 msgr2=0x7f56f819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56f7e00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56f8108d90 0x7f56f819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56f7e00640 1 -- 172.21.15.102:0/1504706469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f81a6c20 con 0x7f56f8109f80 2024-04-26T20:32:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56fd400640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56f810a940 0x7f56f81a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:46.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56f7e00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56f8109f80 0x7f56f819f540 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56e400e9f0 tx=0x7f56e400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:46.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56f5600640 1 -- 172.21.15.102:0/1504706469 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e400cd80 con 0x7f56f8109f80 2024-04-26T20:32:46.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56f5600640 1 -- 172.21.15.102:0/1504706469 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56e40054d0 con 0x7f56f8109f80 2024-04-26T20:32:46.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56fde00640 1 -- 172.21.15.102:0/1504706469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f81a6f10 con 0x7f56f8109f80 2024-04-26T20:32:46.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.009+0000 7f56f5600640 1 -- 172.21.15.102:0/1504706469 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4010690 con 0x7f56f8109f80 2024-04-26T20:32:46.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.010+0000 7f56fde00640 1 -- 172.21.15.102:0/1504706469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f81a7380 con 0x7f56f8109f80 2024-04-26T20:32:46.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.011+0000 7f56fde00640 1 -- 172.21.15.102:0/1504706469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f8110170 con 0x7f56f8109f80 2024-04-26T20:32:46.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.013+0000 7f56f5600640 1 -- 172.21.15.102:0/1504706469 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f56e4004e90 con 0x7f56f8109f80 2024-04-26T20:32:46.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.015+0000 7f56f5600640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f56e00a3bb0 0x7f56e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.015+0000 7f56f5600640 1 -- 172.21.15.102:0/1504706469 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f56e40d0a20 con 0x7f56f8109f80 2024-04-26T20:32:46.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.015+0000 7f56fca00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f56e00a3bb0 0x7f56e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.016+0000 7f56fca00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f56e00a3bb0 0x7f56e00a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009830 tx=0x7f56ec030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:46.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.019+0000 7f56f5600640 1 -- 172.21.15.102:0/1504706469 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f56e409d1c0 con 0x7f56f8109f80 2024-04-26T20:32:46.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:45 smithi112 ceph-mon[32769]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T20:32:46.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:45 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3751487568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:32:46.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:45 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1034038601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:32:46.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.226+0000 7f56fde00640 1 -- 172.21.15.102:0/1504706469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56f80635d0 con 0x7f56f8109f80 2024-04-26T20:32:46.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.228+0000 7f56f5600640 1 -- 172.21.15.102:0/1504706469 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24164 (secure 0 0 0) 0x7f56e409cac0 con 0x7f56f8109f80 2024-04-26T20:32:46.230 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:46.230 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:32:30.827511+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:32:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 -- 172.21.15.102:0/1504706469 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f56e00a3bb0 msgr2=0x7f56e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f56e00a3bb0 0x7f56e00a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009830 tx=0x7f56ec030450 comp rx=0 tx=0).stop 2024-04-26T20:32:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 -- 172.21.15.102:0/1504706469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56f8109f80 msgr2=0x7f56f819f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56f8109f80 0x7f56f819f540 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56e400e9f0 tx=0x7f56e400edf0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 -- 172.21.15.102:0/1504706469 shutdown_connections 2024-04-26T20:32:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f56e00a3bb0 0x7f56e00a6070 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56f810a940 0x7f56f81a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56f8109f80 0x7f56f819f540 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 --2- 172.21.15.102:0/1504706469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56f8108d90 0x7f56f819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 -- 172.21.15.102:0/1504706469 >> 172.21.15.102:0/1504706469 conn(0x7f56f8077d30 msgr2=0x7f56f810d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 -- 172.21.15.102:0/1504706469 shutdown_connections 2024-04-26T20:32:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.230+0000 7f56fde00640 1 -- 172.21.15.102:0/1504706469 wait complete. 2024-04-26T20:32:46.251 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:27:10.527933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:32:23.439237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:32:24.702263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:32:28.348785+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:32:46.251 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:32:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.450+0000 7f8774400640 1 -- 172.21.15.102:0/779199709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876c078ed0 msgr2=0x7f876c079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.450+0000 7f8774400640 1 --2- 172.21.15.102:0/779199709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876c078ed0 0x7f876c079770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8764009ca0 tx=0x7f876402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.450+0000 7f8774400640 1 -- 172.21.15.102:0/779199709 shutdown_connections 2024-04-26T20:32:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.450+0000 7f8774400640 1 --2- 172.21.15.102:0/779199709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876c078ed0 0x7f876c079770 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.450+0000 7f8774400640 1 --2- 172.21.15.102:0/779199709 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f876c078530 0x7f876c078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.450+0000 7f8774400640 1 --2- 172.21.15.102:0/779199709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f876c0772f0 0x7f876c077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.450+0000 7f8774400640 1 -- 172.21.15.102:0/779199709 >> 172.21.15.102:0/779199709 conn(0x7f876c100120 msgr2=0x7f876c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.450+0000 7f8774400640 1 -- 172.21.15.102:0/779199709 shutdown_connections 2024-04-26T20:32:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.450+0000 7f8774400640 1 -- 172.21.15.102:0/779199709 wait complete. 2024-04-26T20:32:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8774400640 1 Processor -- start 2024-04-26T20:32:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8774400640 1 -- start start 2024-04-26T20:32:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8774400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f876c0772f0 0x7f876c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8774400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f876c078530 0x7f876c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8774400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876c078ed0 0x7f876c1a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8774400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f876c114b10 con 0x7f876c078ed0 2024-04-26T20:32:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8774400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f876c114990 con 0x7f876c0772f0 2024-04-26T20:32:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8774400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f876c114c90 con 0x7f876c078530 2024-04-26T20:32:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8772400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876c078ed0 0x7f876c1a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8771a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f876c0772f0 0x7f876c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.451+0000 7f8772400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876c078ed0 0x7f876c1a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51822/0 (socket says 172.21.15.102:51822) 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.452+0000 7f8772400640 1 -- 172.21.15.102:0/3424692293 learned_addr learned my addr 172.21.15.102:0/3424692293 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.452+0000 7f8771000640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f876c078530 0x7f876c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.452+0000 7f8771a00640 1 -- 172.21.15.102:0/3424692293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f876c078530 msgr2=0x7f876c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.452+0000 7f8771a00640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f876c078530 0x7f876c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.452+0000 7f8771a00640 1 -- 172.21.15.102:0/3424692293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876c078ed0 msgr2=0x7f876c1a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.452+0000 7f8771a00640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876c078ed0 0x7f876c1a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.452+0000 7f8771a00640 1 -- 172.21.15.102:0/3424692293 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c1a6dc0 con 0x7f876c0772f0 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.452+0000 7f8771a00640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f876c0772f0 0x7f876c19f1a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f876000e9f0 tx=0x7f876000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.453+0000 7f876a000640 1 -- 172.21.15.102:0/3424692293 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876000ccc0 con 0x7f876c0772f0 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.453+0000 7f876a000640 1 -- 172.21.15.102:0/3424692293 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87600108e0 con 0x7f876c0772f0 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.453+0000 7f876a000640 1 -- 172.21.15.102:0/3424692293 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760018880 con 0x7f876c0772f0 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.453+0000 7f8774400640 1 -- 172.21.15.102:0/3424692293 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876c1a70b0 con 0x7f876c0772f0 2024-04-26T20:32:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.453+0000 7f8774400640 1 -- 172.21.15.102:0/3424692293 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f876c1a7650 con 0x7f876c0772f0 2024-04-26T20:32:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.454+0000 7f8774400640 1 -- 172.21.15.102:0/3424692293 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f876c111500 con 0x7f876c0772f0 2024-04-26T20:32:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.456+0000 7f876a000640 1 -- 172.21.15.102:0/3424692293 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f8760010430 con 0x7f876c0772f0 2024-04-26T20:32:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.457+0000 7f876a000640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f87540a3c30 0x7f87540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.457+0000 7f876a000640 1 -- 172.21.15.102:0/3424692293 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f87600d0d80 con 0x7f876c0772f0 2024-04-26T20:32:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.457+0000 7f8771000640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f87540a3c30 0x7f87540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.458+0000 7f8771000640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f87540a3c30 0x7f87540a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f875c0095f0 tx=0x7f875c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.459+0000 7f876a000640 1 -- 172.21.15.102:0/3424692293 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f876009ce70 con 0x7f876c0772f0 2024-04-26T20:32:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.648+0000 7f8774400640 1 -- 172.21.15.102:0/3424692293 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f876c063530 con 0x7f876c0772f0 2024-04-26T20:32:46.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.649+0000 7f876a000640 1 -- 172.21.15.102:0/3424692293 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f87600a1170 con 0x7f876c0772f0 2024-04-26T20:32:46.650 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-26T20:32:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 -- 172.21.15.102:0/3424692293 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f87540a3c30 msgr2=0x7f87540a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f87540a3c30 0x7f87540a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f875c0095f0 tx=0x7f875c005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 -- 172.21.15.102:0/3424692293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f876c0772f0 msgr2=0x7f876c19f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f876c0772f0 0x7f876c19f1a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f876000e9f0 tx=0x7f876000eeb0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 -- 172.21.15.102:0/3424692293 shutdown_connections 2024-04-26T20:32:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f87540a3c30 0x7f87540a60f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876c078ed0 0x7f876c1a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f876c078530 0x7f876c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 --2- 172.21.15.102:0/3424692293 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f876c0772f0 0x7f876c19f1a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 -- 172.21.15.102:0/3424692293 >> 172.21.15.102:0/3424692293 conn(0x7f876c100120 msgr2=0x7f876c101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 -- 172.21.15.102:0/3424692293 shutdown_connections 2024-04-26T20:32:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.652+0000 7f8774400640 1 -- 172.21.15.102:0/3424692293 wait complete. 2024-04-26T20:32:46.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:32:46.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.868+0000 7fb015c00640 1 -- 172.21.15.102:0/1559950396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb010106620 msgr2=0x7fb01010ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.868+0000 7fb015c00640 1 --2- 172.21.15.102:0/1559950396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb010106620 0x7fb01010ceb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb008009ca0 tx=0x7fb00802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.868+0000 7fb015c00640 1 -- 172.21.15.102:0/1559950396 shutdown_connections 2024-04-26T20:32:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.868+0000 7fb015c00640 1 --2- 172.21.15.102:0/1559950396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb010106620 0x7fb01010ceb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.868+0000 7fb015c00640 1 --2- 172.21.15.102:0/1559950396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb010105c60 0x7fb0101060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.868+0000 7fb015c00640 1 --2- 172.21.15.102:0/1559950396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb010104a70 0x7fb010104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.868+0000 7fb015c00640 1 -- 172.21.15.102:0/1559950396 >> 172.21.15.102:0/1559950396 conn(0x7fb0100fffe0 msgr2=0x7fb010102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.868+0000 7fb015c00640 1 -- 172.21.15.102:0/1559950396 shutdown_connections 2024-04-26T20:32:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.868+0000 7fb015c00640 1 -- 172.21.15.102:0/1559950396 wait complete. 2024-04-26T20:32:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.869+0000 7fb015c00640 1 Processor -- start 2024-04-26T20:32:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb015c00640 1 -- start start 2024-04-26T20:32:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb015c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb010104a70 0x7fb01019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb015c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb010105c60 0x7fb01019b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb015c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb010106620 0x7fb0101a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb015c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb0101105f0 con 0x7fb010106620 2024-04-26T20:32:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb015c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb010110470 con 0x7fb010105c60 2024-04-26T20:32:46.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb015c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb010110770 con 0x7fb010104a70 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb00f400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb010105c60 0x7fb01019b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb00fe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb010104a70 0x7fb01019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb00f400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb010105c60 0x7fb01019b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:37238/0 (socket says 172.21.15.102:37238) 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb00f400640 1 -- 172.21.15.102:0/4263680247 learned_addr learned my addr 172.21.15.102:0/4263680247 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.870+0000 7fb015200640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb010106620 0x7fb0101a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb00f400640 1 -- 172.21.15.102:0/4263680247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb010104a70 msgr2=0x7fb01019ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb00f400640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb010104a70 0x7fb01019ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb00f400640 1 -- 172.21.15.102:0/4263680247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb010106620 msgr2=0x7fb0101a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb00f400640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb010106620 0x7fb0101a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb00f400640 1 -- 172.21.15.102:0/4263680247 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0101a2870 con 0x7fb010105c60 2024-04-26T20:32:46.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb00fe00640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb010104a70 0x7fb01019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb00f400640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb010105c60 0x7fb01019b1a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faffc00e9f0 tx=0x7faffc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb00cc00640 1 -- 172.21.15.102:0/4263680247 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc00cd80 con 0x7fb010105c60 2024-04-26T20:32:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb00cc00640 1 -- 172.21.15.102:0/4263680247 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faffc0054d0 con 0x7fb010105c60 2024-04-26T20:32:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb015c00640 1 -- 172.21.15.102:0/4263680247 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0101a2b60 con 0x7fb010105c60 2024-04-26T20:32:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.871+0000 7fb00cc00640 1 -- 172.21.15.102:0/4263680247 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc010690 con 0x7fb010105c60 2024-04-26T20:32:46.875 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:46 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3124712133' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:32:46.875 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:46 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1504706469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.872+0000 7fb015c00640 1 -- 172.21.15.102:0/4263680247 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0101a3060 con 0x7fb010105c60 2024-04-26T20:32:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.873+0000 7fb015c00640 1 -- 172.21.15.102:0/4263680247 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb01010bd90 con 0x7fb010105c60 2024-04-26T20:32:46.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.874+0000 7fb00cc00640 1 -- 172.21.15.102:0/4263680247 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7faffc004e90 con 0x7fb010105c60 2024-04-26T20:32:46.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.876+0000 7fb00cc00640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faff40a3bb0 0x7faff40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.876+0000 7fb00cc00640 1 -- 172.21.15.102:0/4263680247 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7faffc0d0b20 con 0x7fb010105c60 2024-04-26T20:32:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.876+0000 7fb00fe00640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faff40a3bb0 0x7faff40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:46.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.877+0000 7fb00fe00640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faff40a3bb0 0x7faff40a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb01019c130 tx=0x7fb000005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:46.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:46.880+0000 7fb00cc00640 1 -- 172.21.15.102:0/4263680247 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faffc09d260 con 0x7fb010105c60 2024-04-26T20:32:46.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:46 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3124712133' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:32:46.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:46 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1504706469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:47.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:46 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3124712133' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:32:47.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:46 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1504706469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:47.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.085+0000 7fb015c00640 1 -- 172.21.15.102:0/4263680247 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb010063530 con 0x7fb010105c60 2024-04-26T20:32:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.086+0000 7fb00cc00640 1 -- 172.21.15.102:0/4263680247 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7faffc09cb60 con 0x7fb010105c60 2024-04-26T20:32:47.088 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:32:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 -- 172.21.15.102:0/4263680247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faff40a3bb0 msgr2=0x7faff40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faff40a3bb0 0x7faff40a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb01019c130 tx=0x7fb000005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 -- 172.21.15.102:0/4263680247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb010105c60 msgr2=0x7fb01019b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb010105c60 0x7fb01019b1a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faffc00e9f0 tx=0x7faffc00edf0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 -- 172.21.15.102:0/4263680247 shutdown_connections 2024-04-26T20:32:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faff40a3bb0 0x7faff40a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb010106620 0x7fb0101a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb010105c60 0x7fb01019b1a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 --2- 172.21.15.102:0/4263680247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb010104a70 0x7fb01019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 -- 172.21.15.102:0/4263680247 >> 172.21.15.102:0/4263680247 conn(0x7fb0100fffe0 msgr2=0x7fb010108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 -- 172.21.15.102:0/4263680247 shutdown_connections 2024-04-26T20:32:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.089+0000 7fb015c00640 1 -- 172.21.15.102:0/4263680247 wait complete. 2024-04-26T20:32:47.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:32:47.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.314+0000 7f871c800640 1 -- 172.21.15.102:0/2232345031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8714105d80 msgr2=0x7f8714106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.314+0000 7f871c800640 1 --2- 172.21.15.102:0/2232345031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8714105d80 0x7f8714106200 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8704009900 tx=0x7f870402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.314+0000 7f871c800640 1 -- 172.21.15.102:0/2232345031 shutdown_connections 2024-04-26T20:32:47.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.314+0000 7f871c800640 1 --2- 172.21.15.102:0/2232345031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8714106740 0x7f871410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.314+0000 7f871c800640 1 --2- 172.21.15.102:0/2232345031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8714105d80 0x7f8714106200 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.314+0000 7f871c800640 1 --2- 172.21.15.102:0/2232345031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8714104b90 0x7f8714104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.314+0000 7f871c800640 1 -- 172.21.15.102:0/2232345031 >> 172.21.15.102:0/2232345031 conn(0x7f8714100120 msgr2=0x7f8714102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:47.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.315+0000 7f871c800640 1 -- 172.21.15.102:0/2232345031 shutdown_connections 2024-04-26T20:32:47.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.315+0000 7f871c800640 1 -- 172.21.15.102:0/2232345031 wait complete. 2024-04-26T20:32:47.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.316+0000 7f871c800640 1 Processor -- start 2024-04-26T20:32:47.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.316+0000 7f871c800640 1 -- start start 2024-04-26T20:32:47.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.316+0000 7f871c800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8714104b90 0x7f871419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:47.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.316+0000 7f871c800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8714105d80 0x7f871419b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:47.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.316+0000 7f871c800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8714106740 0x7f87141a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:47.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.316+0000 7f871c800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8714110710 con 0x7f8714104b90 2024-04-26T20:32:47.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.316+0000 7f871c800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8714110590 con 0x7f8714105d80 2024-04-26T20:32:47.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.316+0000 7f871c800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8714110890 con 0x7f8714106740 2024-04-26T20:32:47.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.316+0000 7f8719e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8714104b90 0x7f871419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:47.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.316+0000 7f8719e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8714104b90 0x7f871419ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51864/0 (socket says 172.21.15.102:51864) 2024-04-26T20:32:47.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.317+0000 7f8719e00640 1 -- 172.21.15.102:0/1907020871 learned_addr learned my addr 172.21.15.102:0/1907020871 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:47.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.317+0000 7f8719400640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8714105d80 0x7f871419b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:47.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.317+0000 7f871a800640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8714106740 0x7f87141a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:47.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.317+0000 7f8719e00640 1 -- 172.21.15.102:0/1907020871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8714106740 msgr2=0x7f87141a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.317+0000 7f8719e00640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8714106740 0x7f87141a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.317+0000 7f8719e00640 1 -- 172.21.15.102:0/1907020871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8714105d80 msgr2=0x7f871419b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.317+0000 7f8719e00640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8714105d80 0x7f871419b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.317+0000 7f8719e00640 1 -- 172.21.15.102:0/1907020871 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87141a2980 con 0x7f8714104b90 2024-04-26T20:32:47.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.317+0000 7f8719400640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8714105d80 0x7f871419b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:47.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.317+0000 7f8719e00640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8714104b90 0x7f871419ad60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f870800d770 tx=0x7f870800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:47.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.318+0000 7f870ea00640 1 -- 172.21.15.102:0/1907020871 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708017070 con 0x7f8714104b90 2024-04-26T20:32:47.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.318+0000 7f870ea00640 1 -- 172.21.15.102:0/1907020871 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8708009c30 con 0x7f8714104b90 2024-04-26T20:32:47.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.318+0000 7f870ea00640 1 -- 172.21.15.102:0/1907020871 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708003cc0 con 0x7f8714104b90 2024-04-26T20:32:47.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.318+0000 7f871c800640 1 -- 172.21.15.102:0/1907020871 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87141a2c70 con 0x7f8714104b90 2024-04-26T20:32:47.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.318+0000 7f871c800640 1 -- 172.21.15.102:0/1907020871 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87141a3140 con 0x7f8714104b90 2024-04-26T20:32:47.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.319+0000 7f871c800640 1 -- 172.21.15.102:0/1907020871 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f871410beb0 con 0x7f8714104b90 2024-04-26T20:32:47.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.321+0000 7f870ea00640 1 -- 172.21.15.102:0/1907020871 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f8708003e60 con 0x7f8714104b90 2024-04-26T20:32:47.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.322+0000 7f870ea00640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f86fc0a3bb0 0x7f86fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:47.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.322+0000 7f870ea00640 1 -- 172.21.15.102:0/1907020871 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f87080cff60 con 0x7f8714104b90 2024-04-26T20:32:47.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.323+0000 7f8719400640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f86fc0a3bb0 0x7f86fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:47.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.323+0000 7f8719400640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f86fc0a3bb0 0x7f86fc0a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8704031500 tx=0x7f8704005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:47.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.327+0000 7f870ea00640 1 -- 172.21.15.102:0/1907020871 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8708009de0 con 0x7f8714104b90 2024-04-26T20:32:47.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.530+0000 7f871c800640 1 -- 172.21.15.102:0/1907020871 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f87140635d0 con 0x7f8714104b90 2024-04-26T20:32:47.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.531+0000 7f870ea00640 1 -- 172.21.15.102:0/1907020871 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f870809c090 con 0x7f8714104b90 2024-04-26T20:32:47.532 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-26T20:32:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 -- 172.21.15.102:0/1907020871 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f86fc0a3bb0 msgr2=0x7f86fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f86fc0a3bb0 0x7f86fc0a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8704031500 tx=0x7f8704005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 -- 172.21.15.102:0/1907020871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8714104b90 msgr2=0x7f871419ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8714104b90 0x7f871419ad60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f870800d770 tx=0x7f870800dc30 comp rx=0 tx=0).stop 2024-04-26T20:32:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 -- 172.21.15.102:0/1907020871 shutdown_connections 2024-04-26T20:32:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f86fc0a3bb0 0x7f86fc0a6070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8714106740 0x7f87141a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8714105d80 0x7f871419b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 --2- 172.21.15.102:0/1907020871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8714104b90 0x7f871419ad60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 -- 172.21.15.102:0/1907020871 >> 172.21.15.102:0/1907020871 conn(0x7f8714100120 msgr2=0x7f8714109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 -- 172.21.15.102:0/1907020871 shutdown_connections 2024-04-26T20:32:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.534+0000 7f871c800640 1 -- 172.21.15.102:0/1907020871 wait complete. 2024-04-26T20:32:47.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:32:47.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.759+0000 7f6f9b200640 1 -- 172.21.15.102:0/1696450424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f9c103a30 msgr2=0x7f6f9c103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.759+0000 7f6f9b200640 1 --2- 172.21.15.102:0/1696450424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f9c103a30 0x7f6f9c103eb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009860 tx=0x7f6f8402a340 comp rx=0 tx=0).stop 2024-04-26T20:32:47.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.759+0000 7f6f9b200640 1 -- 172.21.15.102:0/1696450424 shutdown_connections 2024-04-26T20:32:47.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.759+0000 7f6f9b200640 1 --2- 172.21.15.102:0/1696450424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c1043f0 0x7f6f9c10ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.759+0000 7f6f9b200640 1 --2- 172.21.15.102:0/1696450424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f9c103a30 0x7f6f9c103eb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.759+0000 7f6f9b200640 1 --2- 172.21.15.102:0/1696450424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f9c102840 0x7f6f9c102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.759+0000 7f6f9b200640 1 -- 172.21.15.102:0/1696450424 >> 172.21.15.102:0/1696450424 conn(0x7f6f9c0fddd0 msgr2=0x7f6f9c1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:47.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.759+0000 7f6f9b200640 1 -- 172.21.15.102:0/1696450424 shutdown_connections 2024-04-26T20:32:47.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.759+0000 7f6f9b200640 1 -- 172.21.15.102:0/1696450424 wait complete. 2024-04-26T20:32:47.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.760+0000 7f6f9b200640 1 Processor -- start 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.760+0000 7f6f9b200640 1 -- start start 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.760+0000 7f6f9b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c102840 0x7f6f9c19ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.760+0000 7f6f9b200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f9c103a30 0x7f6f9c19b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.760+0000 7f6f9b200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f9c1043f0 0x7f6f9c1a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.760+0000 7f6f9b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c10e350 con 0x7f6f9c102840 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.760+0000 7f6f9b200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c10e1d0 con 0x7f6f9c1043f0 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.760+0000 7f6f9b200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c10e4d0 con 0x7f6f9c103a30 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f99400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f9c103a30 0x7f6f9c19b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f99e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c102840 0x7f6f9c19ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f99400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f9c103a30 0x7f6f9c19b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:36788/0 (socket says 172.21.15.102:36788) 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f99400640 1 -- 172.21.15.102:0/2034965742 learned_addr learned my addr 172.21.15.102:0/2034965742 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f9a800640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f9c1043f0 0x7f6f9c1a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f99400640 1 -- 172.21.15.102:0/2034965742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f9c1043f0 msgr2=0x7f6f9c1a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f99400640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f9c1043f0 0x7f6f9c1a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f99400640 1 -- 172.21.15.102:0/2034965742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c102840 msgr2=0x7f6f9c19ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f99400640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c102840 0x7f6f9c19ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f99400640 1 -- 172.21.15.102:0/2034965742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9c1a28d0 con 0x7f6f9c103a30 2024-04-26T20:32:47.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.761+0000 7f6f99400640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f9c103a30 0x7f6f9c19b1d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f840057c0 tx=0x7f6f84032da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:47.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.762+0000 7f6f92a00640 1 -- 172.21.15.102:0/2034965742 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f84037040 con 0x7f6f9c103a30 2024-04-26T20:32:47.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.762+0000 7f6f9b200640 1 -- 172.21.15.102:0/2034965742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9c1a2b60 con 0x7f6f9c103a30 2024-04-26T20:32:47.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.762+0000 7f6f92a00640 1 -- 172.21.15.102:0/2034965742 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f84030bd0 con 0x7f6f9c103a30 2024-04-26T20:32:47.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.762+0000 7f6f92a00640 1 -- 172.21.15.102:0/2034965742 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f84005280 con 0x7f6f9c103a30 2024-04-26T20:32:47.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.762+0000 7f6f9b200640 1 -- 172.21.15.102:0/2034965742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f9c1a2ff0 con 0x7f6f9c103a30 2024-04-26T20:32:47.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.763+0000 7f6f9b200640 1 -- 172.21.15.102:0/2034965742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f9c1a3220 con 0x7f6f9c103a30 2024-04-26T20:32:47.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.765+0000 7f6f92a00640 1 -- 172.21.15.102:0/2034965742 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f6f84043020 con 0x7f6f9c103a30 2024-04-26T20:32:47.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.766+0000 7f6f92a00640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f800a3bb0 0x7f6f800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:47.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.766+0000 7f6f92a00640 1 -- 172.21.15.102:0/2034965742 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6f840efcc0 con 0x7f6f9c103a30 2024-04-26T20:32:47.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.766+0000 7f6f99e00640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f800a3bb0 0x7f6f800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:47.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.767+0000 7f6f99e00640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f800a3bb0 0x7f6f800a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c19bb60 tx=0x7f6f8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:47.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.771+0000 7f6f92a00640 1 -- 172.21.15.102:0/2034965742 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6f840f6080 con 0x7f6f9c103a30 2024-04-26T20:32:47.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:47 smithi130 ceph-mon[32704]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T20:32:47.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:47 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3424692293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:32:47.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:47 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/4263680247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:32:47.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:47 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1907020871' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:32:47.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:47 smithi102 ceph-mon[28094]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T20:32:47.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:47 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3424692293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:32:47.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:47 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/4263680247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:32:47.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:47 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1907020871' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:32:47.928 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:47 smithi112 ceph-mon[32769]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T20:32:47.928 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:47 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3424692293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:32:47.929 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:47 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/4263680247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:32:47.929 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:47 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1907020871' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:32:47.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.973+0000 7f6f9b200640 1 -- 172.21.15.102:0/2034965742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6f9c1a35b0 con 0x7f6f9c103a30 2024-04-26T20:32:47.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.974+0000 7f6f92a00640 1 -- 172.21.15.102:0/2034965742 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f6f840bc400 con 0x7f6f9c103a30 2024-04-26T20:32:47.975 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:32:47.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.976+0000 7f6f9b200640 1 -- 172.21.15.102:0/2034965742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f800a3bb0 msgr2=0x7f6f800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.976+0000 7f6f9b200640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f800a3bb0 0x7f6f800a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c19bb60 tx=0x7f6f8c005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:47.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.976+0000 7f6f9b200640 1 -- 172.21.15.102:0/2034965742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f9c103a30 msgr2=0x7f6f9c19b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:47.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.976+0000 7f6f9b200640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f9c103a30 0x7f6f9c19b1d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f840057c0 tx=0x7f6f84032da0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.976+0000 7f6f9b200640 1 -- 172.21.15.102:0/2034965742 shutdown_connections 2024-04-26T20:32:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.976+0000 7f6f9b200640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6f800a3bb0 0x7f6f800a6070 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.976+0000 7f6f9b200640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f9c1043f0 0x7f6f9c1a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.976+0000 7f6f9b200640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f9c103a30 0x7f6f9c19b1d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.976+0000 7f6f9b200640 1 --2- 172.21.15.102:0/2034965742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c102840 0x7f6f9c19ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.977+0000 7f6f9b200640 1 -- 172.21.15.102:0/2034965742 >> 172.21.15.102:0/2034965742 conn(0x7f6f9c0fddd0 msgr2=0x7f6f9c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.977+0000 7f6f9b200640 1 -- 172.21.15.102:0/2034965742 shutdown_connections 2024-04-26T20:32:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:47.977+0000 7f6f9b200640 1 -- 172.21.15.102:0/2034965742 wait complete. 2024-04-26T20:32:47.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T20:32:48.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.198+0000 7f9612200640 1 -- 172.21.15.102:0/2104853083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f960c102980 msgr2=0x7f960c102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:48.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.198+0000 7f9612200640 1 --2- 172.21.15.102:0/2104853083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f960c102980 0x7f960c102da0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009900 tx=0x7f95fc02a410 comp rx=0 tx=0).stop 2024-04-26T20:32:48.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.198+0000 7f9612200640 1 -- 172.21.15.102:0/2104853083 shutdown_connections 2024-04-26T20:32:48.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.198+0000 7f9612200640 1 --2- 172.21.15.102:0/2104853083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c104530 0x7f960c10adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.198+0000 7f9612200640 1 --2- 172.21.15.102:0/2104853083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f960c103b70 0x7f960c103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.198+0000 7f9612200640 1 --2- 172.21.15.102:0/2104853083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f960c102980 0x7f960c102da0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.198+0000 7f9612200640 1 -- 172.21.15.102:0/2104853083 >> 172.21.15.102:0/2104853083 conn(0x7f960c0fdf10 msgr2=0x7f960c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:48.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.198+0000 7f9612200640 1 -- 172.21.15.102:0/2104853083 shutdown_connections 2024-04-26T20:32:48.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.198+0000 7f9612200640 1 -- 172.21.15.102:0/2104853083 wait complete. 2024-04-26T20:32:48.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.199+0000 7f9612200640 1 Processor -- start 2024-04-26T20:32:48.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.199+0000 7f9612200640 1 -- start start 2024-04-26T20:32:48.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.199+0000 7f9612200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c102980 0x7f960c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:48.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f9612200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f960c103b70 0x7f960c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f9612200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f960c104530 0x7f960c1a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f9612200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f960c10e4f0 con 0x7f960c102980 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f9612200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f960c10e370 con 0x7f960c103b70 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f9612200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f960c10e670 con 0x7f960c104530 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c102980 0x7f960c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f960c104530 0x7f960c1a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c102980 0x7f960c19adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51910/0 (socket says 172.21.15.102:51910) 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f960c104530 0x7f960c1a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:36810/0 (socket says 172.21.15.102:36810) 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960b400640 1 -- 172.21.15.102:0/805197936 learned_addr learned my addr 172.21.15.102:0/805197936 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960aa00640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f960c103b70 0x7f960c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:48.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960b400640 1 -- 172.21.15.102:0/805197936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f960c104530 msgr2=0x7f960c1a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:48.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960b400640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f960c104530 0x7f960c1a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960b400640 1 -- 172.21.15.102:0/805197936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f960c103b70 msgr2=0x7f960c19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:48.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960b400640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f960c103b70 0x7f960c19b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.200+0000 7f960b400640 1 -- 172.21.15.102:0/805197936 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960c1a29e0 con 0x7f960c102980 2024-04-26T20:32:48.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.201+0000 7f960b400640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c102980 0x7f960c19adc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f95fc031500 tx=0x7f95fc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:48.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.201+0000 7f9603e00640 1 -- 172.21.15.102:0/805197936 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc03f070 con 0x7f960c102980 2024-04-26T20:32:48.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.201+0000 7f9603e00640 1 -- 172.21.15.102:0/805197936 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95fc031c50 con 0x7f960c102980 2024-04-26T20:32:48.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.201+0000 7f9612200640 1 -- 172.21.15.102:0/805197936 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960c1a2c70 con 0x7f960c102980 2024-04-26T20:32:48.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.201+0000 7f9603e00640 1 -- 172.21.15.102:0/805197936 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc035030 con 0x7f960c102980 2024-04-26T20:32:48.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.201+0000 7f9612200640 1 -- 172.21.15.102:0/805197936 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f960c1a3160 con 0x7f960c102980 2024-04-26T20:32:48.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.202+0000 7f9612200640 1 -- 172.21.15.102:0/805197936 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f960c1a3330 con 0x7f960c102980 2024-04-26T20:32:48.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.204+0000 7f9603e00640 1 -- 172.21.15.102:0/805197936 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f95fc042070 con 0x7f960c102980 2024-04-26T20:32:48.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.206+0000 7f9603e00640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95ec0a3bb0 0x7f95ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:48.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.206+0000 7f9603e00640 1 -- 172.21.15.102:0/805197936 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f95fc0f14c0 con 0x7f960c102980 2024-04-26T20:32:48.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.206+0000 7f960aa00640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95ec0a3bb0 0x7f95ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:48.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.207+0000 7f960aa00640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95ec0a3bb0 0x7f95ec0a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f960c19c2d0 tx=0x7f95f4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:48.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.208+0000 7f9603e00640 1 -- 172.21.15.102:0/805197936 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f95fc0bd5b0 con 0x7f960c102980 2024-04-26T20:32:48.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.423+0000 7f9612200640 1 -- 172.21.15.102:0/805197936 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f960c1a3500 con 0x7f960c102980 2024-04-26T20:32:48.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.424+0000 7f9603e00640 1 -- 172.21.15.102:0/805197936 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+602 (secure 0 0 0) 0x7f95fc0c18b0 con 0x7f960c102980 2024-04-26T20:32:48.425 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2611652511 2024-04-27T20:24:28.376337+0000 2024-04-26T20:32:48.425 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/1441707916 2024-04-27T20:24:28.376337+0000 2024-04-26T20:32:48.425 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/1441707916 2024-04-27T20:24:28.376337+0000 2024-04-26T20:32:48.426 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2809626135 2024-04-27T20:24:28.376337+0000 2024-04-26T20:32:48.426 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2372720525 2024-04-27T20:23:53.695696+0000 2024-04-26T20:32:48.426 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2628958579 2024-04-27T20:23:53.695696+0000 2024-04-26T20:32:48.426 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/3976810658 2024-04-27T20:23:53.695696+0000 2024-04-26T20:32:48.426 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/3976810658 2024-04-27T20:23:53.695696+0000 2024-04-26T20:32:48.427 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/3271239188 2024-04-27T20:24:28.376337+0000 2024-04-26T20:32:48.427 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1384283564 2024-04-27T20:23:53.695696+0000 2024-04-26T20:32:48.427 INFO:teuthology.orchestra.run.smithi102.stderr:listed 10 entries 2024-04-26T20:32:48.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 -- 172.21.15.102:0/805197936 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95ec0a3bb0 msgr2=0x7f95ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:48.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95ec0a3bb0 0x7f95ec0a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f960c19c2d0 tx=0x7f95f4009500 comp rx=0 tx=0).stop 2024-04-26T20:32:48.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 -- 172.21.15.102:0/805197936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c102980 msgr2=0x7f960c19adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:48.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c102980 0x7f960c19adc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f95fc031500 tx=0x7f95fc003c10 comp rx=0 tx=0).stop 2024-04-26T20:32:48.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 -- 172.21.15.102:0/805197936 shutdown_connections 2024-04-26T20:32:48.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f95ec0a3bb0 0x7f95ec0a6070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f960c104530 0x7f960c1a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f960c103b70 0x7f960c19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 --2- 172.21.15.102:0/805197936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c102980 0x7f960c19adc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 -- 172.21.15.102:0/805197936 >> 172.21.15.102:0/805197936 conn(0x7f960c0fdf10 msgr2=0x7f960c109340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:48.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 -- 172.21.15.102:0/805197936 shutdown_connections 2024-04-26T20:32:48.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.427+0000 7f9612200640 1 -- 172.21.15.102:0/805197936 wait complete. 2024-04-26T20:32:48.447 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi102.front.sepia.ceph.com... 2024-04-26T20:32:48.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:48.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-26T20:32:48.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:ceph-fuse[71624]: fuse finished with error 0 and tester_r 0 2024-04-26T20:32:48.523 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T20:32:48.545 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-04-26T20:32:48.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:32:48.546 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-26T20:32:48.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:32:48.714 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:48 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2034965742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:32:48.714 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:48 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/805197936' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:32:48.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.846+0000 7fc035000640 1 -- 172.21.15.102:0/1542890099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc030109fa0 msgr2=0x7fc03010a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:48.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.846+0000 7fc035000640 1 --2- 172.21.15.102:0/1542890099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc030109fa0 0x7fc03010a420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc018009900 tx=0x7fc01802a410 comp rx=0 tx=0).stop 2024-04-26T20:32:48.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.846+0000 7fc035000640 1 -- 172.21.15.102:0/1542890099 shutdown_connections 2024-04-26T20:32:48.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.846+0000 7fc035000640 1 --2- 172.21.15.102:0/1542890099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc03010a960 0x7fc0301111f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.846+0000 7fc035000640 1 --2- 172.21.15.102:0/1542890099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc030109fa0 0x7fc03010a420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.846+0000 7fc035000640 1 --2- 172.21.15.102:0/1542890099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc030108db0 0x7fc0301091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.846+0000 7fc035000640 1 -- 172.21.15.102:0/1542890099 >> 172.21.15.102:0/1542890099 conn(0x7fc030077d70 msgr2=0x7fc03007a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:48.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.846+0000 7fc035000640 1 -- 172.21.15.102:0/1542890099 shutdown_connections 2024-04-26T20:32:48.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.846+0000 7fc035000640 1 -- 172.21.15.102:0/1542890099 wait complete. 2024-04-26T20:32:48.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.848+0000 7fc035000640 1 Processor -- start 2024-04-26T20:32:48.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.848+0000 7fc035000640 1 -- start start 2024-04-26T20:32:48.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.848+0000 7fc035000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc030108db0 0x7fc03019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:48.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.848+0000 7fc035000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc030109fa0 0x7fc03019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:48.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.848+0000 7fc035000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc03010a960 0x7fc0301a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:48.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.848+0000 7fc035000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc030114910 con 0x7fc03010a960 2024-04-26T20:32:48.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.848+0000 7fc035000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc030114790 con 0x7fc030109fa0 2024-04-26T20:32:48.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.848+0000 7fc035000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc030114a90 con 0x7fc030108db0 2024-04-26T20:32:48.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc030109fa0 0x7fc03019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:48.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc03010a960 0x7fc0301a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:48.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc030109fa0 0x7fc03019f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:37290/0 (socket says 172.21.15.102:37290) 2024-04-26T20:32:48.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02ea00640 1 -- 172.21.15.102:0/2361385370 learned_addr learned my addr 172.21.15.102:0/2361385370 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:48.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02f400640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc030108db0 0x7fc03019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:48.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02fe00640 1 -- 172.21.15.102:0/2361385370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc030108db0 msgr2=0x7fc03019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:48.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02fe00640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc030108db0 0x7fc03019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02fe00640 1 -- 172.21.15.102:0/2361385370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc030109fa0 msgr2=0x7fc03019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:48.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02fe00640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc030109fa0 0x7fc03019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:48.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02fe00640 1 -- 172.21.15.102:0/2361385370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0301a6ba0 con 0x7fc03010a960 2024-04-26T20:32:48.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.849+0000 7fc02fe00640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc03010a960 0x7fc0301a6530 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc028009fd0 tx=0x7fc02800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:48.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.850+0000 7fc027e00640 1 -- 172.21.15.102:0/2361385370 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02800ce20 con 0x7fc03010a960 2024-04-26T20:32:48.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.850+0000 7fc027e00640 1 -- 172.21.15.102:0/2361385370 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc028005020 con 0x7fc03010a960 2024-04-26T20:32:48.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.850+0000 7fc035000640 1 -- 172.21.15.102:0/2361385370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0301a6e90 con 0x7fc03010a960 2024-04-26T20:32:48.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.850+0000 7fc027e00640 1 -- 172.21.15.102:0/2361385370 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028010690 con 0x7fc03010a960 2024-04-26T20:32:48.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.850+0000 7fc035000640 1 -- 172.21.15.102:0/2361385370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0301a7300 con 0x7fc03010a960 2024-04-26T20:32:48.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.851+0000 7fc035000640 1 -- 172.21.15.102:0/2361385370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0301100d0 con 0x7fc03010a960 2024-04-26T20:32:48.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.853+0000 7fc027e00640 1 -- 172.21.15.102:0/2361385370 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fc0280054d0 con 0x7fc03010a960 2024-04-26T20:32:48.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.854+0000 7fc027e00640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc0140a3bb0 0x7fc0140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:48.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.854+0000 7fc027e00640 1 -- 172.21.15.102:0/2361385370 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc0280d0d10 con 0x7fc03010a960 2024-04-26T20:32:48.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.854+0000 7fc02f400640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc0140a3bb0 0x7fc0140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:48.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.855+0000 7fc02f400640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc0140a3bb0 0x7fc0140a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc030109e00 tx=0x7fc020005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:48.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:48.858+0000 7fc027e00640 1 -- 172.21.15.102:0/2361385370 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc0280d6020 con 0x7fc03010a960 2024-04-26T20:32:48.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:48 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2034965742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:32:48.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:48 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/805197936' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:32:48.945 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:48 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2034965742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:32:48.945 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:48 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/805197936' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:32:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.062+0000 7fc035000640 1 -- 172.21.15.102:0/2361385370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0300635d0 con 0x7fc03010a960 2024-04-26T20:32:49.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.064+0000 7fc027e00640 1 -- 172.21.15.102:0/2361385370 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24164 (secure 0 0 0) 0x7fc02809d450 con 0x7fc03010a960 2024-04-26T20:32:49.066 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:49.066 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:32:30.827511+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:32:49.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 -- 172.21.15.102:0/2361385370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc0140a3bb0 msgr2=0x7fc0140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:49.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc0140a3bb0 0x7fc0140a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc030109e00 tx=0x7fc020005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:49.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 -- 172.21.15.102:0/2361385370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc03010a960 msgr2=0x7fc0301a6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:49.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc03010a960 0x7fc0301a6530 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc028009fd0 tx=0x7fc02800ef10 comp rx=0 tx=0).stop 2024-04-26T20:32:49.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 -- 172.21.15.102:0/2361385370 shutdown_connections 2024-04-26T20:32:49.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc0140a3bb0 0x7fc0140a6070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc03010a960 0x7fc0301a6530 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc030109fa0 0x7fc03019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 --2- 172.21.15.102:0/2361385370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc030108db0 0x7fc03019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 -- 172.21.15.102:0/2361385370 >> 172.21.15.102:0/2361385370 conn(0x7fc030077d70 msgr2=0x7fc03010d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:49.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.067+0000 7fc035000640 1 -- 172.21.15.102:0/2361385370 shutdown_connections 2024-04-26T20:32:49.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.068+0000 7fc035000640 1 -- 172.21.15.102:0/2361385370 wait complete. 2024-04-26T20:32:49.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:32:49.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.290+0000 7f5613e00640 1 -- 172.21.15.102:0/2314362463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56141032b0 msgr2=0x7f56141056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:49.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.290+0000 7f5613e00640 1 --2- 172.21.15.102:0/2314362463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56141032b0 0x7f56141056a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5604009900 tx=0x7f560402a410 comp rx=0 tx=0).stop 2024-04-26T20:32:49.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.290+0000 7f5613e00640 1 -- 172.21.15.102:0/2314362463 shutdown_connections 2024-04-26T20:32:49.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.290+0000 7f5613e00640 1 --2- 172.21.15.102:0/2314362463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5614105be0 0x7f5614107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.290+0000 7f5613e00640 1 --2- 172.21.15.102:0/2314362463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5614108510 0x7f561410a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.290+0000 7f5613e00640 1 --2- 172.21.15.102:0/2314362463 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f56141032b0 0x7f56141056a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.290+0000 7f5613e00640 1 -- 172.21.15.102:0/2314362463 >> 172.21.15.102:0/2314362463 conn(0x7f56140fcbc0 msgr2=0x7f56140ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:49.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.290+0000 7f5613e00640 1 -- 172.21.15.102:0/2314362463 shutdown_connections 2024-04-26T20:32:49.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.290+0000 7f5613e00640 1 -- 172.21.15.102:0/2314362463 wait complete. 2024-04-26T20:32:49.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.291+0000 7f5613e00640 1 Processor -- start 2024-04-26T20:32:49.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.291+0000 7f5613e00640 1 -- start start 2024-04-26T20:32:49.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f5613e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56141032b0 0x7f561419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:49.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f5613e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5614105be0 0x7f561419b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f5613e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5614108510 0x7f56141a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f5613e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f561410e2f0 con 0x7f56141032b0 2024-04-26T20:32:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f5613e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f561410e170 con 0x7f5614105be0 2024-04-26T20:32:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f5613e00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f561410e470 con 0x7f5614108510 2024-04-26T20:32:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f560a000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5614105be0 0x7f561419b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f5612a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56141032b0 0x7f561419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f560a000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5614105be0 0x7f561419b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:37310/0 (socket says 172.21.15.102:37310) 2024-04-26T20:32:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f560a000640 1 -- 172.21.15.102:0/423324510 learned_addr learned my addr 172.21.15.102:0/423324510 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f5613400640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5614108510 0x7f56141a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:49.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f560a000640 1 -- 172.21.15.102:0/423324510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5614108510 msgr2=0x7f56141a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:49.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f560a000640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5614108510 0x7f56141a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f560a000640 1 -- 172.21.15.102:0/423324510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56141032b0 msgr2=0x7f561419aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:49.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.292+0000 7f560a000640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56141032b0 0x7f561419aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.293+0000 7f560a000640 1 -- 172.21.15.102:0/423324510 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56141a27c0 con 0x7f5614105be0 2024-04-26T20:32:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.293+0000 7f560a000640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5614105be0 0x7f561419b0e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00e9c0 tx=0x7f55fc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.293+0000 7f560be00640 1 -- 172.21.15.102:0/423324510 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc00cd80 con 0x7f5614105be0 2024-04-26T20:32:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.293+0000 7f560be00640 1 -- 172.21.15.102:0/423324510 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55fc0054d0 con 0x7f5614105be0 2024-04-26T20:32:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.293+0000 7f560be00640 1 -- 172.21.15.102:0/423324510 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc010430 con 0x7f5614105be0 2024-04-26T20:32:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.293+0000 7f5613e00640 1 -- 172.21.15.102:0/423324510 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56141a2ab0 con 0x7f5614105be0 2024-04-26T20:32:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.293+0000 7f5613e00640 1 -- 172.21.15.102:0/423324510 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56141a3050 con 0x7f5614105be0 2024-04-26T20:32:49.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.295+0000 7f5613e00640 1 -- 172.21.15.102:0/423324510 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f561410a000 con 0x7f5614105be0 2024-04-26T20:32:49.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.296+0000 7f560be00640 1 -- 172.21.15.102:0/423324510 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f55fc004fd0 con 0x7f5614105be0 2024-04-26T20:32:49.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.298+0000 7f560be00640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55f80a3c30 0x7f55f80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:49.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.298+0000 7f560be00640 1 -- 172.21.15.102:0/423324510 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f55fc0d0980 con 0x7f5614105be0 2024-04-26T20:32:49.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.298+0000 7f5612a00640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55f80a3c30 0x7f55f80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:49.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.299+0000 7f5612a00640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55f80a3c30 0x7f55f80a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5604031500 tx=0x7f5604005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:49.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.302+0000 7f560be00640 1 -- 172.21.15.102:0/423324510 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f55fc09d120 con 0x7f5614105be0 2024-04-26T20:32:49.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.505+0000 7f5613e00640 1 -- 172.21.15.102:0/423324510 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5614063530 con 0x7f5614105be0 2024-04-26T20:32:49.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.507+0000 7f560be00640 1 -- 172.21.15.102:0/423324510 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24164 (secure 0 0 0) 0x7f55fc09ca20 con 0x7f5614105be0 2024-04-26T20:32:49.508 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:49.508 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:32:30.827511+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:32:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.509+0000 7f5613e00640 1 -- 172.21.15.102:0/423324510 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55f80a3c30 msgr2=0x7f55f80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.509+0000 7f5613e00640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55f80a3c30 0x7f55f80a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5604031500 tx=0x7f5604005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.509+0000 7f5613e00640 1 -- 172.21.15.102:0/423324510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5614105be0 msgr2=0x7f561419b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.509+0000 7f5613e00640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5614105be0 0x7f561419b0e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00e9c0 tx=0x7f55fc00ee80 comp rx=0 tx=0).stop 2024-04-26T20:32:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.510+0000 7f5613e00640 1 -- 172.21.15.102:0/423324510 shutdown_connections 2024-04-26T20:32:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.510+0000 7f5613e00640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55f80a3c30 0x7f55f80a60f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.510+0000 7f5613e00640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5614108510 0x7f56141a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.510+0000 7f5613e00640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5614105be0 0x7f561419b0e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.510+0000 7f5613e00640 1 --2- 172.21.15.102:0/423324510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56141032b0 0x7f561419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.510+0000 7f5613e00640 1 -- 172.21.15.102:0/423324510 >> 172.21.15.102:0/423324510 conn(0x7f56140fcbc0 msgr2=0x7f5614102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:49.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.510+0000 7f5613e00640 1 -- 172.21.15.102:0/423324510 shutdown_connections 2024-04-26T20:32:49.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.510+0000 7f5613e00640 1 -- 172.21.15.102:0/423324510 wait complete. 2024-04-26T20:32:49.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T20:32:49.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.736+0000 7fc507600640 1 -- 172.21.15.102:0/3549317585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc500105da0 msgr2=0x7fc500106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:49.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.736+0000 7fc507600640 1 --2- 172.21.15.102:0/3549317585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc500105da0 0x7fc500106220 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8009900 tx=0x7fc4e802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.736+0000 7fc507600640 1 -- 172.21.15.102:0/3549317585 shutdown_connections 2024-04-26T20:32:49.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.736+0000 7fc507600640 1 --2- 172.21.15.102:0/3549317585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc500106760 0x7fc50010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.736+0000 7fc507600640 1 --2- 172.21.15.102:0/3549317585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc500105da0 0x7fc500106220 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.736+0000 7fc507600640 1 --2- 172.21.15.102:0/3549317585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc500104bb0 0x7fc500104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.736+0000 7fc507600640 1 -- 172.21.15.102:0/3549317585 >> 172.21.15.102:0/3549317585 conn(0x7fc500100120 msgr2=0x7fc500102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:49.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.737+0000 7fc507600640 1 -- 172.21.15.102:0/3549317585 shutdown_connections 2024-04-26T20:32:49.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.737+0000 7fc507600640 1 -- 172.21.15.102:0/3549317585 wait complete. 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.738+0000 7fc507600640 1 Processor -- start 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.738+0000 7fc507600640 1 -- start start 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.738+0000 7fc507600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc500104bb0 0x7fc50019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.738+0000 7fc507600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc500105da0 0x7fc50019b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.738+0000 7fc507600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc500106760 0x7fc5001a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.738+0000 7fc507600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc500110730 con 0x7fc500105da0 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.738+0000 7fc507600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc5001105b0 con 0x7fc500106760 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.738+0000 7fc507600640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc5001108b0 con 0x7fc500104bb0 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc505600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc500106760 0x7fc5001a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc505600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc500106760 0x7fc5001a2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:37340/0 (socket says 172.21.15.102:37340) 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc505600640 1 -- 172.21.15.102:0/3978836877 learned_addr learned my addr 172.21.15.102:0/3978836877 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc504c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc500104bb0 0x7fc50019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc4ffe00640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc500105da0 0x7fc50019b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:49.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc505600640 1 -- 172.21.15.102:0/3978836877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc500104bb0 msgr2=0x7fc50019ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:49.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc505600640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc500104bb0 0x7fc50019ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc505600640 1 -- 172.21.15.102:0/3978836877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc500105da0 msgr2=0x7fc50019b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:49.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc505600640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc500105da0 0x7fc50019b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:49.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc505600640 1 -- 172.21.15.102:0/3978836877 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5001a2a80 con 0x7fc500106760 2024-04-26T20:32:49.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc505600640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc500106760 0x7fc5001a2410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800e9b0 tx=0x7fc4f800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:49.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.739+0000 7fc4fd600640 1 -- 172.21.15.102:0/3978836877 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f800cd80 con 0x7fc500106760 2024-04-26T20:32:49.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.740+0000 7fc4fd600640 1 -- 172.21.15.102:0/3978836877 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4f80035f0 con 0x7fc500106760 2024-04-26T20:32:49.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.740+0000 7fc507600640 1 -- 172.21.15.102:0/3978836877 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5001a2d70 con 0x7fc500106760 2024-04-26T20:32:49.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.740+0000 7fc4fd600640 1 -- 172.21.15.102:0/3978836877 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8010690 con 0x7fc500106760 2024-04-26T20:32:49.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.740+0000 7fc507600640 1 -- 172.21.15.102:0/3978836877 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5001a3230 con 0x7fc500106760 2024-04-26T20:32:49.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.741+0000 7fc507600640 1 -- 172.21.15.102:0/3978836877 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc50010bf50 con 0x7fc500106760 2024-04-26T20:32:49.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.743+0000 7fc4fd600640 1 -- 172.21.15.102:0/3978836877 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fc4f8004020 con 0x7fc500106760 2024-04-26T20:32:49.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.744+0000 7fc4fd600640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4e40a3c30 0x7fc4e40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:49.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.744+0000 7fc4fd600640 1 -- 172.21.15.102:0/3978836877 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc4f80d08d0 con 0x7fc500106760 2024-04-26T20:32:49.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.744+0000 7fc504c00640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4e40a3c30 0x7fc4e40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:49.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.745+0000 7fc504c00640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4e40a3c30 0x7fc4e40a60f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc500105c00 tx=0x7fc4f0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:49.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.748+0000 7fc4fd600640 1 -- 172.21.15.102:0/3978836877 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc4f809d010 con 0x7fc500106760 2024-04-26T20:32:49.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:49 smithi130 ceph-mon[32704]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:32:49.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:49 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2361385370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:49.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:49 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/423324510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:49.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:49 smithi102 ceph-mon[28094]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:32:49.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:49 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2361385370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:49.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:49 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/423324510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:49.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:49.993+0000 7fc507600640 1 -- 172.21.15.102:0/3978836877 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc5000635d0 con 0x7fc500106760 2024-04-26T20:32:50.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:49 smithi112 ceph-mon[32769]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:32:50.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:49 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2361385370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:50.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:49 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/423324510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:50.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.630+0000 7fc4fd600640 1 -- 172.21.15.102:0/3978836877 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7fc4f809c910 con 0x7fc500106760 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.633+0000 7fc507600640 1 -- 172.21.15.102:0/3978836877 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4e40a3c30 msgr2=0x7fc4e40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.633+0000 7fc507600640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4e40a3c30 0x7fc4e40a60f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc500105c00 tx=0x7fc4f0009500 comp rx=0 tx=0).stop 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.633+0000 7fc507600640 1 -- 172.21.15.102:0/3978836877 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc500106760 msgr2=0x7fc5001a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.633+0000 7fc507600640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc500106760 0x7fc5001a2410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800e9b0 tx=0x7fc4f800ee70 comp rx=0 tx=0).stop 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.633+0000 7fc507600640 1 -- 172.21.15.102:0/3978836877 shutdown_connections 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.633+0000 7fc507600640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4e40a3c30 0x7fc4e40a60f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.633+0000 7fc507600640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc500105da0 0x7fc50019b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.633+0000 7fc507600640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc500106760 0x7fc5001a2410 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.634+0000 7fc507600640 1 --2- 172.21.15.102:0/3978836877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc500104bb0 0x7fc50019ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.634+0000 7fc507600640 1 -- 172.21.15.102:0/3978836877 >> 172.21.15.102:0/3978836877 conn(0x7fc500100120 msgr2=0x7fc500109120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.634+0000 7fc507600640 1 -- 172.21.15.102:0/3978836877 shutdown_connections 2024-04-26T20:32:50.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.634+0000 7fc507600640 1 -- 172.21.15.102:0/3978836877 wait complete. 2024-04-26T20:32:50.657 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-26T20:32:50.658 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-26T20:32:50.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:50 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3978836877' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T20:32:50.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:50 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T20:32:50.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.874+0000 7f0447400640 1 -- 172.21.15.102:0/453962859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f044010a180 msgr2=0x7f044010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:50.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.874+0000 7f0447400640 1 --2- 172.21.15.102:0/453962859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f044010a180 0x7f044010a600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f042c009860 tx=0x7f042c02a340 comp rx=0 tx=0).stop 2024-04-26T20:32:50.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.874+0000 7f0447400640 1 -- 172.21.15.102:0/453962859 shutdown_connections 2024-04-26T20:32:50.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.874+0000 7f0447400640 1 --2- 172.21.15.102:0/453962859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f044010ab40 0x7f04401113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:50.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.874+0000 7f0447400640 1 --2- 172.21.15.102:0/453962859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f044010a180 0x7f044010a600 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:50.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.874+0000 7f0447400640 1 --2- 172.21.15.102:0/453962859 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0440108f90 0x7f04401093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.874+0000 7f0447400640 1 -- 172.21.15.102:0/453962859 >> 172.21.15.102:0/453962859 conn(0x7f0440077f30 msgr2=0x7f044007a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.874+0000 7f0447400640 1 -- 172.21.15.102:0/453962859 shutdown_connections 2024-04-26T20:32:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.874+0000 7f0447400640 1 -- 172.21.15.102:0/453962859 wait complete. 2024-04-26T20:32:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.875+0000 7f0447400640 1 Processor -- start 2024-04-26T20:32:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.875+0000 7f0447400640 1 -- start start 2024-04-26T20:32:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.875+0000 7f0447400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0440108f90 0x7f044019f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.875+0000 7f0447400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f044010a180 0x7f044019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.875+0000 7f0447400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f044010ab40 0x7f04401a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.875+0000 7f0447400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0440114ad0 con 0x7f044010a180 2024-04-26T20:32:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.875+0000 7f0447400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0440114950 con 0x7f0440108f90 2024-04-26T20:32:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.875+0000 7f0447400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0440114c50 con 0x7f044010ab40 2024-04-26T20:32:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0444a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0440108f90 0x7f044019f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0444a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0440108f90 0x7f044019f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:37350/0 (socket says 172.21.15.102:37350) 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f043fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f044010a180 0x7f044019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0444a00640 1 -- 172.21.15.102:0/3781286921 learned_addr learned my addr 172.21.15.102:0/3781286921 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0445400640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f044010ab40 0x7f04401a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0444a00640 1 -- 172.21.15.102:0/3781286921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f044010ab40 msgr2=0x7f04401a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0444a00640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f044010ab40 0x7f04401a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0444a00640 1 -- 172.21.15.102:0/3781286921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f044010a180 msgr2=0x7f044019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0444a00640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f044010a180 0x7f044019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0444a00640 1 -- 172.21.15.102:0/3781286921 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04401a6dc0 con 0x7f0440108f90 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0445400640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f044010ab40 0x7f04401a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.876+0000 7f0444a00640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0440108f90 0x7f044019f240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f043400e9f0 tx=0x7f043400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.877+0000 7f043d600640 1 -- 172.21.15.102:0/3781286921 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043400ccc0 con 0x7f0440108f90 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.877+0000 7f0447400640 1 -- 172.21.15.102:0/3781286921 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04401a70b0 con 0x7f0440108f90 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.877+0000 7f043d600640 1 -- 172.21.15.102:0/3781286921 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04340035f0 con 0x7f0440108f90 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.877+0000 7f043d600640 1 -- 172.21.15.102:0/3781286921 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043400ce60 con 0x7f0440108f90 2024-04-26T20:32:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.877+0000 7f0447400640 1 -- 172.21.15.102:0/3781286921 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04401a75d0 con 0x7f0440108f90 2024-04-26T20:32:50.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.878+0000 7f0447400640 1 -- 172.21.15.102:0/3781286921 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04401102d0 con 0x7f0440108f90 2024-04-26T20:32:50.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.880+0000 7f043d600640 1 -- 172.21.15.102:0/3781286921 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f0434004000 con 0x7f0440108f90 2024-04-26T20:32:50.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.881+0000 7f043d600640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f04280a3c30 0x7f04280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:50.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.881+0000 7f043d600640 1 -- 172.21.15.102:0/3781286921 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f04340d0950 con 0x7f0440108f90 2024-04-26T20:32:50.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.881+0000 7f043fe00640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f04280a3c30 0x7f04280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.882+0000 7f043fe00640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f04280a3c30 0x7f04280a60f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f042c02a930 tx=0x7f042c030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:50.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:50.885+0000 7f043d600640 1 -- 172.21.15.102:0/3781286921 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f043409d090 con 0x7f0440108f90 2024-04-26T20:32:50.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:50 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3978836877' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T20:32:50.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:50 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T20:32:51.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:50 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3978836877' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T20:32:51.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:50 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T20:32:51.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.129+0000 7f0447400640 1 -- 172.21.15.102:0/3781286921 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f0440063530 con 0x7f0440108f90 2024-04-26T20:32:51.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.633+0000 7f043d600640 1 -- 172.21.15.102:0/3781286921 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f043409c990 con 0x7f0440108f90 2024-04-26T20:32:51.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.636+0000 7f0447400640 1 -- 172.21.15.102:0/3781286921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f04280a3c30 msgr2=0x7f04280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:51.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.636+0000 7f0447400640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f04280a3c30 0x7f04280a60f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f042c02a930 tx=0x7f042c030450 comp rx=0 tx=0).stop 2024-04-26T20:32:51.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.636+0000 7f0447400640 1 -- 172.21.15.102:0/3781286921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0440108f90 msgr2=0x7f044019f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:51.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.636+0000 7f0447400640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0440108f90 0x7f044019f240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f043400e9f0 tx=0x7f043400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.636+0000 7f0447400640 1 -- 172.21.15.102:0/3781286921 shutdown_connections 2024-04-26T20:32:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.636+0000 7f0447400640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f04280a3c30 0x7f04280a60f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.636+0000 7f0447400640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f044010a180 0x7f044019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.636+0000 7f0447400640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f044010ab40 0x7f04401a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.636+0000 7f0447400640 1 --2- 172.21.15.102:0/3781286921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0440108f90 0x7f044019f240 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.636+0000 7f0447400640 1 -- 172.21.15.102:0/3781286921 >> 172.21.15.102:0/3781286921 conn(0x7f0440077f30 msgr2=0x7f044010d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.637+0000 7f0447400640 1 -- 172.21.15.102:0/3781286921 shutdown_connections 2024-04-26T20:32:51.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.637+0000 7f0447400640 1 -- 172.21.15.102:0/3781286921 wait complete. 2024-04-26T20:32:51.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-26T20:32:51.718 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:51 smithi102 ceph-mon[28094]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:32:51.718 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:51 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T20:32:51.718 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:51 smithi102 ceph-mon[28094]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:51.718 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:51 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3781286921' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-26T20:32:51.718 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:51 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-26T20:32:51.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.845+0000 7fe693000640 1 -- 172.21.15.102:0/1822195956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe68c10ab40 msgr2=0x7fe68c1113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:51.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.845+0000 7fe693000640 1 --2- 172.21.15.102:0/1822195956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe68c10ab40 0x7fe68c1113d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe684009ca0 tx=0x7fe68402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.845+0000 7fe693000640 1 -- 172.21.15.102:0/1822195956 shutdown_connections 2024-04-26T20:32:51.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.845+0000 7fe693000640 1 --2- 172.21.15.102:0/1822195956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe68c10ab40 0x7fe68c1113d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.845+0000 7fe693000640 1 --2- 172.21.15.102:0/1822195956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe68c10a180 0x7fe68c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.845+0000 7fe693000640 1 --2- 172.21.15.102:0/1822195956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe68c108f90 0x7fe68c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.845+0000 7fe693000640 1 -- 172.21.15.102:0/1822195956 >> 172.21.15.102:0/1822195956 conn(0x7fe68c077f30 msgr2=0x7fe68c07a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:51.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.845+0000 7fe693000640 1 -- 172.21.15.102:0/1822195956 shutdown_connections 2024-04-26T20:32:51.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.845+0000 7fe693000640 1 -- 172.21.15.102:0/1822195956 wait complete. 2024-04-26T20:32:51.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.846+0000 7fe693000640 1 Processor -- start 2024-04-26T20:32:51.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.846+0000 7fe693000640 1 -- start start 2024-04-26T20:32:51.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.846+0000 7fe693000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe68c108f90 0x7fe68c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe693000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe68c10a180 0x7fe68c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe693000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe68c10ab40 0x7fe68c1a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe693000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c114b20 con 0x7fe68c10ab40 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe693000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c1149a0 con 0x7fe68c108f90 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe693000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c114ca0 con 0x7fe68c10a180 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe691000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe68c10ab40 0x7fe68c1a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe683e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe68c10a180 0x7fe68c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe691000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe68c10ab40 0x7fe68c1a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52022/0 (socket says 172.21.15.102:52022) 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe691000640 1 -- 172.21.15.102:0/373821694 learned_addr learned my addr 172.21.15.102:0/373821694 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe68be00640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe68c108f90 0x7fe68c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe683e00640 1 -- 172.21.15.102:0/373821694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe68c108f90 msgr2=0x7fe68c19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe683e00640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe68c108f90 0x7fe68c19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe683e00640 1 -- 172.21.15.102:0/373821694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe68c10ab40 msgr2=0x7fe68c1a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe683e00640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe68c10ab40 0x7fe68c1a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.847+0000 7fe683e00640 1 -- 172.21.15.102:0/373821694 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c1a6dc0 con 0x7fe68c10a180 2024-04-26T20:32:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.848+0000 7fe683e00640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe68c10a180 0x7fe68c19f6e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe674005fd0 tx=0x7fe67400dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.848+0000 7fe689600640 1 -- 172.21.15.102:0/373821694 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674009c30 con 0x7fe68c10a180 2024-04-26T20:32:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.848+0000 7fe689600640 1 -- 172.21.15.102:0/373821694 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe674009dd0 con 0x7fe68c10a180 2024-04-26T20:32:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.848+0000 7fe693000640 1 -- 172.21.15.102:0/373821694 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68c1a70b0 con 0x7fe68c10a180 2024-04-26T20:32:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.848+0000 7fe689600640 1 -- 172.21.15.102:0/373821694 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674010460 con 0x7fe68c10a180 2024-04-26T20:32:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.848+0000 7fe693000640 1 -- 172.21.15.102:0/373821694 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe68c1a7600 con 0x7fe68c10a180 2024-04-26T20:32:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.849+0000 7fe693000640 1 -- 172.21.15.102:0/373821694 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68c1102b0 con 0x7fe68c10a180 2024-04-26T20:32:51.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.851+0000 7fe689600640 1 -- 172.21.15.102:0/373821694 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe674003e80 con 0x7fe68c10a180 2024-04-26T20:32:51.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.853+0000 7fe689600640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6700a3bb0 0x7fe6700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:51.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.853+0000 7fe689600640 1 -- 172.21.15.102:0/373821694 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe6740d0560 con 0x7fe68c10a180 2024-04-26T20:32:51.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.853+0000 7fe68be00640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6700a3bb0 0x7fe6700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:51.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.853+0000 7fe68be00640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6700a3bb0 0x7fe6700a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe68c109fe0 tx=0x7fe67c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:51.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:51.855+0000 7fe689600640 1 -- 172.21.15.102:0/373821694 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe674017020 con 0x7fe68c10a180 2024-04-26T20:32:52.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:51 smithi112 ceph-mon[32769]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:32:52.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:51 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T20:32:52.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:51 smithi112 ceph-mon[32769]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:52.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:51 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3781286921' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-26T20:32:52.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:51 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-26T20:32:52.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.100+0000 7fe693000640 1 -- 172.21.15.102:0/373821694 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fe68c02dad0 con 0x7fe68c10a180 2024-04-26T20:32:52.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:51 smithi130 ceph-mon[32704]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:32:52.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:51 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T20:32:52.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:51 smithi130 ceph-mon[32704]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:52.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:51 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3781286921' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-26T20:32:52.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:51 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-26T20:32:52.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.645+0000 7fe689600640 1 -- 172.21.15.102:0/373821694 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7fe67409c650 con 0x7fe68c10a180 2024-04-26T20:32:52.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.648+0000 7fe693000640 1 -- 172.21.15.102:0/373821694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6700a3bb0 msgr2=0x7fe6700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:52.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.648+0000 7fe693000640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6700a3bb0 0x7fe6700a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe68c109fe0 tx=0x7fe67c005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:52.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.648+0000 7fe693000640 1 -- 172.21.15.102:0/373821694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe68c10a180 msgr2=0x7fe68c19f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:52.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.648+0000 7fe693000640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe68c10a180 0x7fe68c19f6e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe674005fd0 tx=0x7fe67400dde0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.649+0000 7fe693000640 1 -- 172.21.15.102:0/373821694 shutdown_connections 2024-04-26T20:32:52.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.649+0000 7fe693000640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe6700a3bb0 0x7fe6700a6070 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.649+0000 7fe693000640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe68c10ab40 0x7fe68c1a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.649+0000 7fe693000640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe68c10a180 0x7fe68c19f6e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.649+0000 7fe693000640 1 --2- 172.21.15.102:0/373821694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe68c108f90 0x7fe68c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.649+0000 7fe693000640 1 -- 172.21.15.102:0/373821694 >> 172.21.15.102:0/373821694 conn(0x7fe68c077f30 msgr2=0x7fe68c10d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:52.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.649+0000 7fe693000640 1 -- 172.21.15.102:0/373821694 shutdown_connections 2024-04-26T20:32:52.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.649+0000 7fe693000640 1 -- 172.21.15.102:0/373821694 wait complete. 2024-04-26T20:32:52.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-26T20:32:52.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.863+0000 7fa173400640 1 -- 172.21.15.102:0/3571187698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c102970 msgr2=0x7fa16c102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.863+0000 7fa173400640 1 --2- 172.21.15.102:0/3571187698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c102970 0x7fa16c102d90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa164009ca0 tx=0x7fa16402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.863+0000 7fa173400640 1 -- 172.21.15.102:0/3571187698 shutdown_connections 2024-04-26T20:32:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.863+0000 7fa173400640 1 --2- 172.21.15.102:0/3571187698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa16c104520 0x7fa16c10adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.863+0000 7fa173400640 1 --2- 172.21.15.102:0/3571187698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa16c103b60 0x7fa16c103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.863+0000 7fa173400640 1 --2- 172.21.15.102:0/3571187698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c102970 0x7fa16c102d90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.863+0000 7fa173400640 1 -- 172.21.15.102:0/3571187698 >> 172.21.15.102:0/3571187698 conn(0x7fa16c0fdee0 msgr2=0x7fa16c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.863+0000 7fa173400640 1 -- 172.21.15.102:0/3571187698 shutdown_connections 2024-04-26T20:32:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.863+0000 7fa173400640 1 -- 172.21.15.102:0/3571187698 wait complete. 2024-04-26T20:32:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.864+0000 7fa173400640 1 Processor -- start 2024-04-26T20:32:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa173400640 1 -- start start 2024-04-26T20:32:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa173400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa16c102970 0x7fa16c0766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa173400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c103b60 0x7fa16c074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa173400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa16c104520 0x7fa16c075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa173400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c10e490 con 0x7fa16c103b60 2024-04-26T20:32:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa173400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c10e310 con 0x7fa16c102970 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa173400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c10e610 con 0x7fa16c104520 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa170a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa16c102970 0x7fa16c0766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa170a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa16c102970 0x7fa16c0766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:37384/0 (socket says 172.21.15.102:37384) 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa170a00640 1 -- 172.21.15.102:0/2360009694 learned_addr learned my addr 172.21.15.102:0/2360009694 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa16be00640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c103b60 0x7fa16c074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.865+0000 7fa171400640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa16c104520 0x7fa16c075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa170a00640 1 -- 172.21.15.102:0/2360009694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa16c104520 msgr2=0x7fa16c075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa170a00640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa16c104520 0x7fa16c075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa170a00640 1 -- 172.21.15.102:0/2360009694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c103b60 msgr2=0x7fa16c074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa170a00640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c103b60 0x7fa16c074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa170a00640 1 -- 172.21.15.102:0/2360009694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c075af0 con 0x7fa16c102970 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa171400640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa16c104520 0x7fa16c075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa170a00640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa16c102970 0x7fa16c0766e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa164009430 tx=0x7fa1640316e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:52.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa169600640 1 -- 172.21.15.102:0/2360009694 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164003b70 con 0x7fa16c102970 2024-04-26T20:32:52.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa173400640 1 -- 172.21.15.102:0/2360009694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16c1a2d40 con 0x7fa16c102970 2024-04-26T20:32:52.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa169600640 1 -- 172.21.15.102:0/2360009694 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa164003d10 con 0x7fa16c102970 2024-04-26T20:32:52.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa169600640 1 -- 172.21.15.102:0/2360009694 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16402fb60 con 0x7fa16c102970 2024-04-26T20:32:52.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.866+0000 7fa173400640 1 -- 172.21.15.102:0/2360009694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa16c1a3180 con 0x7fa16c102970 2024-04-26T20:32:52.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.868+0000 7fa173400640 1 -- 172.21.15.102:0/2360009694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa16c109e00 con 0x7fa16c102970 2024-04-26T20:32:52.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.869+0000 7fa169600640 1 -- 172.21.15.102:0/2360009694 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fa16402fd00 con 0x7fa16c102970 2024-04-26T20:32:52.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.871+0000 7fa169600640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1500a3bb0 0x7fa1500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:52.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.871+0000 7fa169600640 1 -- 172.21.15.102:0/2360009694 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa1640f1b40 con 0x7fa16c102970 2024-04-26T20:32:52.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.871+0000 7fa16be00640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1500a3bb0 0x7fa1500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:52.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.872+0000 7fa16be00640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1500a3bb0 0x7fa1500a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa16c076450 tx=0x7fa154005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:52.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:52.879+0000 7fa169600640 1 -- 172.21.15.102:0/2360009694 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa1640be280 con 0x7fa16c102970 2024-04-26T20:32:53.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:52 smithi112 ceph-mon[32769]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T20:32:53.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:52 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-26T20:32:53.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:52 smithi112 ceph-mon[32769]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T20:32:53.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:52 smithi112 ceph-mon[32769]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:53.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:52 smithi112 ceph-mon[32769]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-26T20:32:53.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:52 smithi112 ceph-mon[32769]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T20:32:53.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:52 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/373821694' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:32:53.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:52 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:32:53.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.110+0000 7fa173400640 1 -- 172.21.15.102:0/2360009694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fa16c063530 con 0x7fa16c102970 2024-04-26T20:32:53.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:52 smithi130 ceph-mon[32704]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T20:32:53.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:52 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-26T20:32:53.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:52 smithi130 ceph-mon[32704]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T20:32:53.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:52 smithi130 ceph-mon[32704]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:53.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:52 smithi130 ceph-mon[32704]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-26T20:32:53.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:52 smithi130 ceph-mon[32704]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T20:32:53.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:52 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/373821694' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:32:53.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:52 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:32:53.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:52 smithi102 ceph-mon[28094]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T20:32:53.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:52 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-26T20:32:53.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:52 smithi102 ceph-mon[28094]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T20:32:53.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:52 smithi102 ceph-mon[28094]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T20:32:53.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:52 smithi102 ceph-mon[28094]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-26T20:32:53.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:52 smithi102 ceph-mon[28094]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T20:32:53.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:52 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/373821694' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:32:53.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:52 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T20:32:53.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.659+0000 7fa169600640 1 -- 172.21.15.102:0/2360009694 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7fa1640bdb80 con 0x7fa16c102970 2024-04-26T20:32:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 -- 172.21.15.102:0/2360009694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1500a3bb0 msgr2=0x7fa1500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1500a3bb0 0x7fa1500a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa16c076450 tx=0x7fa154005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 -- 172.21.15.102:0/2360009694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa16c102970 msgr2=0x7fa16c0766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa16c102970 0x7fa16c0766e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa164009430 tx=0x7fa1640316e0 comp rx=0 tx=0).stop 2024-04-26T20:32:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 -- 172.21.15.102:0/2360009694 shutdown_connections 2024-04-26T20:32:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1500a3bb0 0x7fa1500a6070 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa16c103b60 0x7fa16c074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa16c104520 0x7fa16c075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 --2- 172.21.15.102:0/2360009694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa16c102970 0x7fa16c0766e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:53.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 -- 172.21.15.102:0/2360009694 >> 172.21.15.102:0/2360009694 conn(0x7fa16c0fdee0 msgr2=0x7fa16c106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:53.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 -- 172.21.15.102:0/2360009694 shutdown_connections 2024-04-26T20:32:53.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.664+0000 7fa173400640 1 -- 172.21.15.102:0/2360009694 wait complete. 2024-04-26T20:32:53.695 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-04-26T20:32:53.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-26T20:32:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.883+0000 7fdfea400640 1 -- 172.21.15.102:0/1473572121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfe4102860 msgr2=0x7fdfe4102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.883+0000 7fdfea400640 1 --2- 172.21.15.102:0/1473572121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfe4102860 0x7fdfe4102c80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009900 tx=0x7fdfd802a410 comp rx=0 tx=0).stop 2024-04-26T20:32:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.884+0000 7fdfea400640 1 -- 172.21.15.102:0/1473572121 shutdown_connections 2024-04-26T20:32:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.884+0000 7fdfea400640 1 --2- 172.21.15.102:0/1473572121 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4104410 0x7fdfe410aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.884+0000 7fdfea400640 1 --2- 172.21.15.102:0/1473572121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfe4103a50 0x7fdfe4103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:53.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.884+0000 7fdfea400640 1 --2- 172.21.15.102:0/1473572121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfe4102860 0x7fdfe4102c80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:53.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.884+0000 7fdfea400640 1 -- 172.21.15.102:0/1473572121 >> 172.21.15.102:0/1473572121 conn(0x7fdfe40fddd0 msgr2=0x7fdfe4100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:53.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.884+0000 7fdfea400640 1 -- 172.21.15.102:0/1473572121 shutdown_connections 2024-04-26T20:32:53.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.884+0000 7fdfea400640 1 -- 172.21.15.102:0/1473572121 wait complete. 2024-04-26T20:32:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfea400640 1 Processor -- start 2024-04-26T20:32:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfea400640 1 -- start start 2024-04-26T20:32:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfea400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfe4102860 0x7fdfe419acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:53.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfea400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfe4103a50 0x7fdfe419b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfea400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4104410 0x7fdfe41a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfea400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe410e3d0 con 0x7fdfe4103a50 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfea400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe410e250 con 0x7fdfe4104410 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfea400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe410e550 con 0x7fdfe4102860 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfe3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfe4103a50 0x7fdfe419b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfe3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfe4103a50 0x7fdfe419b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52060/0 (socket says 172.21.15.102:52060) 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfe3e00640 1 -- 172.21.15.102:0/3719853053 learned_addr learned my addr 172.21.15.102:0/3719853053 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.885+0000 7fdfe9a00640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4104410 0x7fdfe41a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe9000640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfe4102860 0x7fdfe419acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe9a00640 1 -- 172.21.15.102:0/3719853053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfe4102860 msgr2=0x7fdfe419acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe9a00640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfe4102860 0x7fdfe419acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe9a00640 1 -- 172.21.15.102:0/3719853053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfe4103a50 msgr2=0x7fdfe419b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe9a00640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfe4103a50 0x7fdfe419b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe9a00640 1 -- 172.21.15.102:0/3719853053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe41a2900 con 0x7fdfe4104410 2024-04-26T20:32:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe3e00640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfe4103a50 0x7fdfe419b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe9a00640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4104410 0x7fdfe41a2260 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00ec50 tx=0x7fdfdc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe1600640 1 -- 172.21.15.102:0/3719853053 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc003b70 con 0x7fdfe4104410 2024-04-26T20:32:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe1600640 1 -- 172.21.15.102:0/3719853053 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfdc003d10 con 0x7fdfe4104410 2024-04-26T20:32:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfea400640 1 -- 172.21.15.102:0/3719853053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe41a2bf0 con 0x7fdfe4104410 2024-04-26T20:32:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.886+0000 7fdfe1600640 1 -- 172.21.15.102:0/3719853053 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc010830 con 0x7fdfe4104410 2024-04-26T20:32:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.887+0000 7fdfea400640 1 -- 172.21.15.102:0/3719853053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe41a30c0 con 0x7fdfe4104410 2024-04-26T20:32:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.888+0000 7fdfea400640 1 -- 172.21.15.102:0/3719853053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfe41a32f0 con 0x7fdfe4104410 2024-04-26T20:32:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.889+0000 7fdfe1600640 1 -- 172.21.15.102:0/3719853053 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fdfdc003ec0 con 0x7fdfe4104410 2024-04-26T20:32:53.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.891+0000 7fdfe1600640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdfcc0a3bb0 0x7fdfcc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.891+0000 7fdfe1600640 1 -- 172.21.15.102:0/3719853053 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdfdc0d0c80 con 0x7fdfe4104410 2024-04-26T20:32:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.891+0000 7fdfe9000640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdfcc0a3bb0 0x7fdfcc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:53.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.891+0000 7fdfe9000640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdfcc0a3bb0 0x7fdfcc0a6070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8005ab0 tx=0x7fdfd8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:53.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:53.895+0000 7fdfe1600640 1 -- 172.21.15.102:0/3719853053 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdfdc09d3c0 con 0x7fdfe4104410 2024-04-26T20:32:54.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:53 smithi112 ceph-mon[32769]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:32:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:53 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T20:32:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:53 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] up:active 2024-04-26T20:32:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:53 smithi112 ceph-mon[32769]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T20:32:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:53 smithi112 ceph-mon[32769]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-26T20:32:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:53 smithi112 ceph-mon[32769]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-26T20:32:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:53 smithi112 ceph-mon[32769]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T20:32:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:53 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2360009694' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T20:32:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:53 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T20:32:54.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:53 smithi130 ceph-mon[32704]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:32:54.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:53 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T20:32:54.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:53 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] up:active 2024-04-26T20:32:54.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:53 smithi130 ceph-mon[32704]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T20:32:54.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:53 smithi130 ceph-mon[32704]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-26T20:32:54.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:53 smithi130 ceph-mon[32704]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-26T20:32:54.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:53 smithi130 ceph-mon[32704]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T20:32:54.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:53 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2360009694' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T20:32:54.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:53 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T20:32:54.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.123+0000 7fdfea400640 1 -- 172.21.15.102:0/3719853053 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fdfe41a34c0 con 0x7fdfcc0a3bb0 2024-04-26T20:32:54.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:53 smithi102 ceph-mon[28094]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T20:32:54.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:53 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T20:32:54.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:53 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] up:active 2024-04-26T20:32:54.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:53 smithi102 ceph-mon[28094]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T20:32:54.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:53 smithi102 ceph-mon[28094]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-26T20:32:54.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:53 smithi102 ceph-mon[28094]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-26T20:32:54.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:53 smithi102 ceph-mon[28094]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T20:32:54.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:53 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2360009694' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T20:32:54.134 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:53 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T20:32:54.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.146+0000 7fdfe1600640 1 -- 172.21.15.102:0/3719853053 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdfe41a34c0 con 0x7fdfcc0a3bb0 2024-04-26T20:32:54.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 -- 172.21.15.102:0/3719853053 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdfcc0a3bb0 msgr2=0x7fdfcc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:54.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdfcc0a3bb0 0x7fdfcc0a6070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8005ab0 tx=0x7fdfd8005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:54.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 -- 172.21.15.102:0/3719853053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4104410 msgr2=0x7fdfe41a2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4104410 0x7fdfe41a2260 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00ec50 tx=0x7fdfdc00c6a0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 -- 172.21.15.102:0/3719853053 shutdown_connections 2024-04-26T20:32:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fdfcc0a3bb0 0x7fdfcc0a6070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdfe4103a50 0x7fdfe419b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfe4104410 0x7fdfe41a2260 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 --2- 172.21.15.102:0/3719853053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfe4102860 0x7fdfe419acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 -- 172.21.15.102:0/3719853053 >> 172.21.15.102:0/3719853053 conn(0x7fdfe40fddd0 msgr2=0x7fdfe4109220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 -- 172.21.15.102:0/3719853053 shutdown_connections 2024-04-26T20:32:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.149+0000 7fdfea400640 1 -- 172.21.15.102:0/3719853053 wait complete. 2024-04-26T20:32:54.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T20:32:54.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.353+0000 7f209d800640 1 -- 172.21.15.102:0/2151208701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098103330 msgr2=0x7f2098105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:54.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.353+0000 7f209d800640 1 --2- 172.21.15.102:0/2151208701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098103330 0x7f2098105720 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2088009900 tx=0x7f208802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.353+0000 7f209d800640 1 -- 172.21.15.102:0/2151208701 shutdown_connections 2024-04-26T20:32:54.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.353+0000 7f209d800640 1 --2- 172.21.15.102:0/2151208701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2098108590 0x7f209810a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.353+0000 7f209d800640 1 --2- 172.21.15.102:0/2151208701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2098105c60 0x7f2098108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.353+0000 7f209d800640 1 --2- 172.21.15.102:0/2151208701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098103330 0x7f2098105720 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.353+0000 7f209d800640 1 -- 172.21.15.102:0/2151208701 >> 172.21.15.102:0/2151208701 conn(0x7f20980fcce0 msgr2=0x7f20980ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:54.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.354+0000 7f209d800640 1 -- 172.21.15.102:0/2151208701 shutdown_connections 2024-04-26T20:32:54.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.354+0000 7f209d800640 1 -- 172.21.15.102:0/2151208701 wait complete. 2024-04-26T20:32:54.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.355+0000 7f209d800640 1 Processor -- start 2024-04-26T20:32:54.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.355+0000 7f209d800640 1 -- start start 2024-04-26T20:32:54.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.355+0000 7f209d800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2098103330 0x7f2098198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:54.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.355+0000 7f209d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2098105c60 0x7f2098199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:54.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.355+0000 7f209d800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098108590 0x7f20981a00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:54.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.355+0000 7f209d800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f209810e3f0 con 0x7f2098105c60 2024-04-26T20:32:54.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.355+0000 7f209d800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f209810e270 con 0x7f2098108590 2024-04-26T20:32:54.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.355+0000 7f209d800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f209810e570 con 0x7f2098103330 2024-04-26T20:32:54.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f2097e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2098103330 0x7f2098198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:54.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f209ce00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098108590 0x7f20981a00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:54.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f2097e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2098103330 0x7f2098198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:36980/0 (socket says 172.21.15.102:36980) 2024-04-26T20:32:54.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f2097e00640 1 -- 172.21.15.102:0/775344529 learned_addr learned my addr 172.21.15.102:0/775344529 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:54.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f2097400640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2098105c60 0x7f2098199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:54.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f209ce00640 1 -- 172.21.15.102:0/775344529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2098103330 msgr2=0x7f2098198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:54.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f209ce00640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2098103330 0x7f2098198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f209ce00640 1 -- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2098105c60 msgr2=0x7f2098199030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:54.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f209ce00640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2098105c60 0x7f2098199030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f209ce00640 1 -- 172.21.15.102:0/775344529 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20981a0710 con 0x7f2098108590 2024-04-26T20:32:54.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f2097400640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2098105c60 0x7f2098199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:54.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.356+0000 7f209ce00640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098108590 0x7f20981a00a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2090009fd0 tx=0x7f209000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:54.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.357+0000 7f2094c00640 1 -- 172.21.15.102:0/775344529 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209000ce20 con 0x7f2098108590 2024-04-26T20:32:54.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.357+0000 7f2094c00640 1 -- 172.21.15.102:0/775344529 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2090005020 con 0x7f2098108590 2024-04-26T20:32:54.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.357+0000 7f2094c00640 1 -- 172.21.15.102:0/775344529 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2090010690 con 0x7f2098108590 2024-04-26T20:32:54.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.357+0000 7f209d800640 1 -- 172.21.15.102:0/775344529 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20981a0a00 con 0x7f2098108590 2024-04-26T20:32:54.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.357+0000 7f209d800640 1 -- 172.21.15.102:0/775344529 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20981a0f00 con 0x7f2098108590 2024-04-26T20:32:54.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.358+0000 7f209d800640 1 -- 172.21.15.102:0/775344529 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20980fe7d0 con 0x7f2098108590 2024-04-26T20:32:54.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.360+0000 7f2094c00640 1 -- 172.21.15.102:0/775344529 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f20900054d0 con 0x7f2098108590 2024-04-26T20:32:54.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.362+0000 7f2094c00640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f207c0a3c30 0x7f207c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:54.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.362+0000 7f2094c00640 1 -- 172.21.15.102:0/775344529 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f20900d0d90 con 0x7f2098108590 2024-04-26T20:32:54.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.362+0000 7f2097e00640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f207c0a3c30 0x7f207c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:54.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.363+0000 7f2097e00640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f207c0a3c30 0x7f207c0a60f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f20880096c0 tx=0x7f2088005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:54.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.366+0000 7f2094c00640 1 -- 172.21.15.102:0/775344529 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f209009d4d0 con 0x7f2098108590 2024-04-26T20:32:54.398 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:32:54.130+0000 7fc2ba400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T20:32:54.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:32:54.130+0000 7fc2ba400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T20:32:54.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:32:54.130+0000 7fc2ba400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T20:32:54.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:32:54.130+0000 7fc2ba400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T20:32:54.399 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:32:54.130+0000 7fc2ba400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T20:32:54.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.605+0000 7f209d800640 1 -- 172.21.15.102:0/775344529 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2098062d70 con 0x7f207c0a3c30 2024-04-26T20:32:54.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.643+0000 7f2094c00640 1 -- 172.21.15.102:0/775344529 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2098062d70 con 0x7f207c0a3c30 2024-04-26T20:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 -- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f207c0a3c30 msgr2=0x7f207c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f207c0a3c30 0x7f207c0a60f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f20880096c0 tx=0x7f2088005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 -- 172.21.15.102:0/775344529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098108590 msgr2=0x7f20981a00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098108590 0x7f20981a00a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2090009fd0 tx=0x7f209000ef10 comp rx=0 tx=0).stop 2024-04-26T20:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 -- 172.21.15.102:0/775344529 shutdown_connections 2024-04-26T20:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f207c0a3c30 0x7f207c0a60f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2098105c60 0x7f2098199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2098108590 0x7f20981a00a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 --2- 172.21.15.102:0/775344529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2098103330 0x7f2098198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 -- 172.21.15.102:0/775344529 >> 172.21.15.102:0/775344529 conn(0x7f20980fcce0 msgr2=0x7f2098103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:54.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 -- 172.21.15.102:0/775344529 shutdown_connections 2024-04-26T20:32:54.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.646+0000 7f209d800640 1 -- 172.21.15.102:0/775344529 wait complete. 2024-04-26T20:32:54.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T20:32:54.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.853+0000 7f31f2400640 1 -- 172.21.15.102:0/298017556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec106620 msgr2=0x7f31ec10ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:54.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.853+0000 7f31f2400640 1 --2- 172.21.15.102:0/298017556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec106620 0x7f31ec10ceb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009ca0 tx=0x7f31e402a8b0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.854+0000 7f31f2400640 1 -- 172.21.15.102:0/298017556 shutdown_connections 2024-04-26T20:32:54.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.854+0000 7f31f2400640 1 --2- 172.21.15.102:0/298017556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec106620 0x7f31ec10ceb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.854+0000 7f31f2400640 1 --2- 172.21.15.102:0/298017556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31ec105c60 0x7f31ec1060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.854+0000 7f31f2400640 1 --2- 172.21.15.102:0/298017556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31ec104a70 0x7f31ec104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.854+0000 7f31f2400640 1 -- 172.21.15.102:0/298017556 >> 172.21.15.102:0/298017556 conn(0x7f31ec0fffe0 msgr2=0x7f31ec102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.854+0000 7f31f2400640 1 -- 172.21.15.102:0/298017556 shutdown_connections 2024-04-26T20:32:54.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.854+0000 7f31f2400640 1 -- 172.21.15.102:0/298017556 wait complete. 2024-04-26T20:32:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f2400640 1 Processor -- start 2024-04-26T20:32:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f2400640 1 -- start start 2024-04-26T20:32:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f2400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31ec104a70 0x7f31ec19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f2400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec105c60 0x7f31ec19b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f2400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31ec106620 0x7f31ec1a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f2400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f31ec1105d0 con 0x7f31ec104a70 2024-04-26T20:32:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f2400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f31ec110450 con 0x7f31ec105c60 2024-04-26T20:32:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f2400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f31ec110750 con 0x7f31ec106620 2024-04-26T20:32:54.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31ec104a70 0x7f31ec19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec105c60 0x7f31ec19b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec105c60 0x7f31ec19b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:37460/0 (socket says 172.21.15.102:37460) 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31ec104a70 0x7f31ec19abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52116/0 (socket says 172.21.15.102:52116) 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31ebe00640 1 -- 172.21.15.102:0/3528594265 learned_addr learned my addr 172.21.15.102:0/3528594265 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.855+0000 7f31f1a00640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31ec106620 0x7f31ec1a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31ebe00640 1 -- 172.21.15.102:0/3528594265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31ec106620 msgr2=0x7f31ec1a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31ebe00640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31ec106620 0x7f31ec1a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31ebe00640 1 -- 172.21.15.102:0/3528594265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31ec104a70 msgr2=0x7f31ec19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31ebe00640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31ec104a70 0x7f31ec19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31ebe00640 1 -- 172.21.15.102:0/3528594265 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31ec1a2810 con 0x7f31ec105c60 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31ebe00640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec105c60 0x7f31ec19b130 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f31e000d950 tx=0x7f31e000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31e9600640 1 -- 172.21.15.102:0/3528594265 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e0017030 con 0x7f31ec105c60 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31e9600640 1 -- 172.21.15.102:0/3528594265 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31e0004020 con 0x7f31ec105c60 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31e9600640 1 -- 172.21.15.102:0/3528594265 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e00047a0 con 0x7f31ec105c60 2024-04-26T20:32:54.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31f2400640 1 -- 172.21.15.102:0/3528594265 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31ec1a2b00 con 0x7f31ec105c60 2024-04-26T20:32:54.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.856+0000 7f31f2400640 1 -- 172.21.15.102:0/3528594265 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31ec1a3000 con 0x7f31ec105c60 2024-04-26T20:32:54.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.857+0000 7f31f2400640 1 -- 172.21.15.102:0/3528594265 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31ec10bd90 con 0x7f31ec105c60 2024-04-26T20:32:54.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.859+0000 7f31e9600640 1 -- 172.21.15.102:0/3528594265 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f31e0009a30 con 0x7f31ec105c60 2024-04-26T20:32:54.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.860+0000 7f31e9600640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f31d40a3bb0 0x7f31d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:54.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.861+0000 7f31e9600640 1 -- 172.21.15.102:0/3528594265 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f31e00d0110 con 0x7f31ec105c60 2024-04-26T20:32:54.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.861+0000 7f31f1000640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f31d40a3bb0 0x7f31d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:54.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.861+0000 7f31f1000640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f31d40a3bb0 0x7f31d40a6070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f31ec105ac0 tx=0x7f31dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:54.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:54.865+0000 7f31e9600640 1 -- 172.21.15.102:0/3528594265 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f31e009c850 con 0x7f31ec105c60 2024-04-26T20:32:54.947 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:54 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T20:32:54.947 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:54 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] up:active 2024-04-26T20:32:54.947 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:54 smithi130 ceph-mon[32704]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-26T20:32:54.948 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:54 smithi130 ceph-mon[32704]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-26T20:32:54.948 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:54 smithi130 ceph-mon[32704]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-04-26T20:32:54.948 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:54 smithi130 ceph-mon[32704]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-26T20:32:55.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:54 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T20:32:55.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:54 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] up:active 2024-04-26T20:32:55.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:54 smithi112 ceph-mon[32769]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-26T20:32:55.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:54 smithi112 ceph-mon[32769]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-26T20:32:55.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:54 smithi112 ceph-mon[32769]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-04-26T20:32:55.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:54 smithi112 ceph-mon[32769]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-26T20:32:55.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.080+0000 7f31f2400640 1 -- 172.21.15.102:0/3528594265 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f31ec062d70 con 0x7f31d40a3bb0 2024-04-26T20:32:55.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.094+0000 7f31e9600640 1 -- 172.21.15.102:0/3528594265 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f31ec062d70 con 0x7f31d40a3bb0 2024-04-26T20:32:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:/volumes/qa/sv_0/fa4cdc1d-e1d4-47dc-8559-e607bc0b58d9 2024-04-26T20:32:55.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 -- 172.21.15.102:0/3528594265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f31d40a3bb0 msgr2=0x7f31d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:55.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f31d40a3bb0 0x7f31d40a6070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f31ec105ac0 tx=0x7f31dc005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:55.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 -- 172.21.15.102:0/3528594265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec105c60 msgr2=0x7f31ec19b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:55.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec105c60 0x7f31ec19b130 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f31e000d950 tx=0x7f31e000de10 comp rx=0 tx=0).stop 2024-04-26T20:32:55.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 -- 172.21.15.102:0/3528594265 shutdown_connections 2024-04-26T20:32:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f31d40a3bb0 0x7f31d40a6070 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31ec106620 0x7f31ec1a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31ec105c60 0x7f31ec19b130 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 --2- 172.21.15.102:0/3528594265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31ec104a70 0x7f31ec19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 -- 172.21.15.102:0/3528594265 >> 172.21.15.102:0/3528594265 conn(0x7f31ec0fffe0 msgr2=0x7f31ec108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 -- 172.21.15.102:0/3528594265 shutdown_connections 2024-04-26T20:32:55.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.096+0000 7f31f2400640 1 -- 172.21.15.102:0/3528594265 wait complete. 2024-04-26T20:32:55.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T20:32:55.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] up:active 2024-04-26T20:32:55.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-mon[28094]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-26T20:32:55.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-mon[28094]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-26T20:32:55.112 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-mon[28094]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-04-26T20:32:55.113 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:54 smithi102 ceph-mon[28094]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-26T20:32:55.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T20:32:55.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.312+0000 7f6f17200640 1 -- 172.21.15.102:0/74396639 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f1810a780 msgr2=0x7f6f1810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:55.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.312+0000 7f6f17200640 1 --2- 172.21.15.102:0/74396639 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f1810a780 0x7f6f1810cc10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009ca0 tx=0x7f6f1002a8c0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.312+0000 7f6f17200640 1 -- 172.21.15.102:0/74396639 shutdown_connections 2024-04-26T20:32:55.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.312+0000 7f6f17200640 1 --2- 172.21.15.102:0/74396639 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f18075390 0x7f6f18075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.312+0000 7f6f17200640 1 --2- 172.21.15.102:0/74396639 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f1810a780 0x7f6f1810cc10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.312+0000 7f6f17200640 1 --2- 172.21.15.102:0/74396639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f18076970 0x7f6f18074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.312+0000 7f6f17200640 1 -- 172.21.15.102:0/74396639 >> 172.21.15.102:0/74396639 conn(0x7f6f180fde30 msgr2=0x7f6f18100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:55.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.312+0000 7f6f17200640 1 -- 172.21.15.102:0/74396639 shutdown_connections 2024-04-26T20:32:55.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.313+0000 7f6f17200640 1 -- 172.21.15.102:0/74396639 wait complete. 2024-04-26T20:32:55.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f17200640 1 Processor -- start 2024-04-26T20:32:55.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f17200640 1 -- start start 2024-04-26T20:32:55.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f17200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f18075390 0x7f6f1819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:55.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f17200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f18076970 0x7f6f1819b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:55.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f17200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f1810a780 0x7f6f181a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:55.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f17200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6f18110560 con 0x7f6f1810a780 2024-04-26T20:32:55.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f17200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6f181103e0 con 0x7f6f18076970 2024-04-26T20:32:55.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f17200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6f181106e0 con 0x7f6f18075390 2024-04-26T20:32:55.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f15400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f18076970 0x7f6f1819b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f15400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f18076970 0x7f6f1819b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:36698/0 (socket says 172.21.15.102:36698) 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f16800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f1810a780 0x7f6f181a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.314+0000 7f6f15400640 1 -- 172.21.15.102:0/3395976265 learned_addr learned my addr 172.21.15.102:0/3395976265 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f15e00640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f18075390 0x7f6f1819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f15400640 1 -- 172.21.15.102:0/3395976265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f18075390 msgr2=0x7f6f1819ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f15400640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f18075390 0x7f6f1819ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f15400640 1 -- 172.21.15.102:0/3395976265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f1810a780 msgr2=0x7f6f181a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f15400640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f1810a780 0x7f6f181a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f15400640 1 -- 172.21.15.102:0/3395976265 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f181a2830 con 0x7f6f18076970 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f16800640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f1810a780 0x7f6f181a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f15400640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f18076970 0x7f6f1819b160 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400e9f0 tx=0x7f6f0400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:55.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f0ea00640 1 -- 172.21.15.102:0/3395976265 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0400cd80 con 0x7f6f18076970 2024-04-26T20:32:55.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f17200640 1 -- 172.21.15.102:0/3395976265 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f181a2b20 con 0x7f6f18076970 2024-04-26T20:32:55.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f0ea00640 1 -- 172.21.15.102:0/3395976265 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f040108e0 con 0x7f6f18076970 2024-04-26T20:32:55.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f0ea00640 1 -- 172.21.15.102:0/3395976265 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04018980 con 0x7f6f18076970 2024-04-26T20:32:55.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.315+0000 7f6f17200640 1 -- 172.21.15.102:0/3395976265 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f181a3070 con 0x7f6f18076970 2024-04-26T20:32:55.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.316+0000 7f6f17200640 1 -- 172.21.15.102:0/3395976265 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f181094b0 con 0x7f6f18076970 2024-04-26T20:32:55.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.319+0000 7f6f0ea00640 1 -- 172.21.15.102:0/3395976265 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f6f04010430 con 0x7f6f18076970 2024-04-26T20:32:55.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.320+0000 7f6f0ea00640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6efc0a3bb0 0x7f6efc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:55.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.321+0000 7f6f0ea00640 1 -- 172.21.15.102:0/3395976265 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6f040d0d90 con 0x7f6f18076970 2024-04-26T20:32:55.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.321+0000 7f6f15e00640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6efc0a3bb0 0x7f6efc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:55.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.321+0000 7f6f15e00640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6efc0a3bb0 0x7f6efc0a6070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6f1819c0f0 tx=0x7f6f08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:55.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.322+0000 7f6f0ea00640 1 -- 172.21.15.102:0/3395976265 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6f0409d4d0 con 0x7f6f18076970 2024-04-26T20:32:55.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.545+0000 7f6f17200640 1 -- 172.21.15.102:0/3395976265 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f6f18062e00 con 0x7f6efc0a3bb0 2024-04-26T20:32:55.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.578+0000 7f6f0ea00640 1 -- 172.21.15.102:0/3395976265 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6f18062e00 con 0x7f6efc0a3bb0 2024-04-26T20:32:55.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.580+0000 7f6f17200640 1 -- 172.21.15.102:0/3395976265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6efc0a3bb0 msgr2=0x7f6efc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:55.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.580+0000 7f6f17200640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6efc0a3bb0 0x7f6efc0a6070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6f1819c0f0 tx=0x7f6f08005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:55.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.580+0000 7f6f17200640 1 -- 172.21.15.102:0/3395976265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f18076970 msgr2=0x7f6f1819b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:55.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.580+0000 7f6f17200640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f18076970 0x7f6f1819b160 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400e9f0 tx=0x7f6f0400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.580+0000 7f6f17200640 1 -- 172.21.15.102:0/3395976265 shutdown_connections 2024-04-26T20:32:55.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.580+0000 7f6f17200640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6efc0a3bb0 0x7f6efc0a6070 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.580+0000 7f6f17200640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f1810a780 0x7f6f181a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.580+0000 7f6f17200640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f18076970 0x7f6f1819b160 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.580+0000 7f6f17200640 1 --2- 172.21.15.102:0/3395976265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f18075390 0x7f6f1819ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.580+0000 7f6f17200640 1 -- 172.21.15.102:0/3395976265 >> 172.21.15.102:0/3395976265 conn(0x7f6f180fde30 msgr2=0x7f6f18106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:55.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.581+0000 7f6f17200640 1 -- 172.21.15.102:0/3395976265 shutdown_connections 2024-04-26T20:32:55.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.581+0000 7f6f17200640 1 -- 172.21.15.102:0/3395976265 wait complete. 2024-04-26T20:32:55.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T20:32:55.779 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:55 smithi102 ceph-mon[28094]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:55.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:55 smithi102 ceph-mon[28094]: pgmap v332: 129 pgs: 129 active+clean; 583 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-26T20:32:55.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:55 smithi102 ceph-mon[28094]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:55.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:55 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] up:active 2024-04-26T20:32:55.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:55 smithi102 ceph-mon[28094]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-26T20:32:55.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:55 smithi102 ceph-mon[28094]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T20:32:55.780 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:55 smithi102 ceph-mon[28094]: Cluster is now healthy 2024-04-26T20:32:55.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:55 smithi102 ceph-mon[28094]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-04-26T20:32:55.781 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:55 smithi102 ceph-mon[28094]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-26T20:32:55.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:55 smithi102 ceph-mon[28094]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-26T20:32:55.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.795+0000 7f4cb8400640 1 -- 172.21.15.102:0/3218128357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cb0103b70 msgr2=0x7f4cb0103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:55.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.795+0000 7f4cb8400640 1 --2- 172.21.15.102:0/3218128357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cb0103b70 0x7f4cb0103ff0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009900 tx=0x7f4c9c02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.795+0000 7f4cb8400640 1 -- 172.21.15.102:0/3218128357 shutdown_connections 2024-04-26T20:32:55.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.795+0000 7f4cb8400640 1 --2- 172.21.15.102:0/3218128357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cb0104530 0x7f4cb010adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.795+0000 7f4cb8400640 1 --2- 172.21.15.102:0/3218128357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cb0103b70 0x7f4cb0103ff0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.795+0000 7f4cb8400640 1 --2- 172.21.15.102:0/3218128357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cb0102980 0x7f4cb0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.795+0000 7f4cb8400640 1 -- 172.21.15.102:0/3218128357 >> 172.21.15.102:0/3218128357 conn(0x7f4cb00fdf10 msgr2=0x7f4cb0100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:55.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.795+0000 7f4cb8400640 1 -- 172.21.15.102:0/3218128357 shutdown_connections 2024-04-26T20:32:55.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.795+0000 7f4cb8400640 1 -- 172.21.15.102:0/3218128357 wait complete. 2024-04-26T20:32:55.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.796+0000 7f4cb8400640 1 Processor -- start 2024-04-26T20:32:55.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.796+0000 7f4cb8400640 1 -- start start 2024-04-26T20:32:55.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.796+0000 7f4cb8400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cb0102980 0x7f4cb019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:55.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.796+0000 7f4cb8400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cb0103b70 0x7f4cb019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:55.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.796+0000 7f4cb8400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cb0104530 0x7f4cb01a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:55.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.796+0000 7f4cb8400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb010e500 con 0x7f4cb0103b70 2024-04-26T20:32:55.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.796+0000 7f4cb8400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb010e380 con 0x7f4cb0102980 2024-04-26T20:32:55.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.796+0000 7f4cb8400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb010e680 con 0x7f4cb0104530 2024-04-26T20:32:55.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb5a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cb0102980 0x7f4cb019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:55.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb5000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cb0103b70 0x7f4cb019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:55.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb5a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cb0102980 0x7f4cb019adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:36714/0 (socket says 172.21.15.102:36714) 2024-04-26T20:32:55.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb5a00640 1 -- 172.21.15.102:0/3017524856 learned_addr learned my addr 172.21.15.102:0/3017524856 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:55.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb6400640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cb0104530 0x7f4cb01a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb5a00640 1 -- 172.21.15.102:0/3017524856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cb0104530 msgr2=0x7f4cb01a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb5a00640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cb0104530 0x7f4cb01a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb5a00640 1 -- 172.21.15.102:0/3017524856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cb0103b70 msgr2=0x7f4cb019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb5a00640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cb0103b70 0x7f4cb019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb5a00640 1 -- 172.21.15.102:0/3017524856 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb01a29e0 con 0x7f4cb0102980 2024-04-26T20:32:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb6400640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cb0104530 0x7f4cb01a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:32:55.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb5a00640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cb0102980 0x7f4cb019adc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400ebc0 tx=0x7f4ca400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:55.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cae000640 1 -- 172.21.15.102:0/3017524856 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4003b70 con 0x7f4cb0102980 2024-04-26T20:32:55.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cae000640 1 -- 172.21.15.102:0/3017524856 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ca4003d10 con 0x7f4cb0102980 2024-04-26T20:32:55.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cb8400640 1 -- 172.21.15.102:0/3017524856 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb01a2cd0 con 0x7f4cb0102980 2024-04-26T20:32:55.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.797+0000 7f4cae000640 1 -- 172.21.15.102:0/3017524856 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4019800 con 0x7f4cb0102980 2024-04-26T20:32:55.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.798+0000 7f4cb8400640 1 -- 172.21.15.102:0/3017524856 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb01a3190 con 0x7f4cb0102980 2024-04-26T20:32:55.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.799+0000 7f4cb8400640 1 -- 172.21.15.102:0/3017524856 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cb01a3420 con 0x7f4cb0102980 2024-04-26T20:32:55.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.800+0000 7f4cae000640 1 -- 172.21.15.102:0/3017524856 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f4ca40199a0 con 0x7f4cb0102980 2024-04-26T20:32:55.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.802+0000 7f4cae000640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4c980a3cc0 0x7f4c980a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:55.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.802+0000 7f4cae000640 1 -- 172.21.15.102:0/3017524856 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4ca40d10e0 con 0x7f4cb0102980 2024-04-26T20:32:55.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.802+0000 7f4cb5000640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4c980a3cc0 0x7f4c980a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:55.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.803+0000 7f4cb5000640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4c980a3cc0 0x7f4c980a6180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c031500 tx=0x7f4c9c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:55.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:55.806+0000 7f4cae000640 1 -- 172.21.15.102:0/3017524856 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4ca409d6c0 con 0x7f4cb0102980 2024-04-26T20:32:56.026 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b[62857]: 2024-04-26T20:32:55.676+0000 7f74d1749700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b[62857]: 2024-04-26T20:32:55.676+0000 7f74d1749700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b[62857]: 2024-04-26T20:32:55.676+0000 7f74d1749700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-mon[32769]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-mon[32769]: pgmap v332: 129 pgs: 129 active+clean; 583 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-mon[32769]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] up:active 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-mon[32769]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-mon[32769]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-mon[32769]: Cluster is now healthy 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-mon[32769]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-04-26T20:32:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-mon[32769]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-26T20:32:56.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:55 smithi112 ceph-mon[32769]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-26T20:32:56.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.041+0000 7f4cb8400640 1 -- 172.21.15.102:0/3017524856 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f4cb0076280 con 0x7f4c980a3cc0 2024-04-26T20:32:56.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.052+0000 7f4cae000640 1 -- 172.21.15.102:0/3017524856 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4cb0076280 con 0x7f4c980a3cc0 2024-04-26T20:32:56.053 INFO:teuthology.orchestra.run.smithi102.stdout:/volumes/qa/sv_1/7baa4043-cba2-4147-ad16-9a164859bb63 2024-04-26T20:32:56.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 -- 172.21.15.102:0/3017524856 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4c980a3cc0 msgr2=0x7f4c980a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4c980a3cc0 0x7f4c980a6180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c031500 tx=0x7f4c9c005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:56.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 -- 172.21.15.102:0/3017524856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cb0102980 msgr2=0x7f4cb019adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cb0102980 0x7f4cb019adc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400ebc0 tx=0x7f4ca400c6a0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 -- 172.21.15.102:0/3017524856 shutdown_connections 2024-04-26T20:32:56.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4c980a3cc0 0x7f4c980a6180 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cb0103b70 0x7f4cb019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cb0104530 0x7f4cb01a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 --2- 172.21.15.102:0/3017524856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cb0102980 0x7f4cb019adc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 -- 172.21.15.102:0/3017524856 >> 172.21.15.102:0/3017524856 conn(0x7f4cb00fdf10 msgr2=0x7f4cb0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:56.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 -- 172.21.15.102:0/3017524856 shutdown_connections 2024-04-26T20:32:56.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.055+0000 7f4cb8400640 1 -- 172.21.15.102:0/3017524856 wait complete. 2024-04-26T20:32:56.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:32:56.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:55 smithi130 ceph-mon[32704]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:56.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:55 smithi130 ceph-mon[32704]: pgmap v332: 129 pgs: 129 active+clean; 583 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 3 op/s 2024-04-26T20:32:56.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:55 smithi130 ceph-mon[32704]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:56.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:55 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] up:active 2024-04-26T20:32:56.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:55 smithi130 ceph-mon[32704]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-26T20:32:56.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:55 smithi130 ceph-mon[32704]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T20:32:56.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:55 smithi130 ceph-mon[32704]: Cluster is now healthy 2024-04-26T20:32:56.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:55 smithi130 ceph-mon[32704]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-04-26T20:32:56.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:55 smithi130 ceph-mon[32704]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-26T20:32:56.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:55 smithi130 ceph-mon[32704]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-26T20:32:56.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.257+0000 7f6a3f000640 1 -- 172.21.15.102:0/851212159 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38104320 msgr2=0x7f6a3810abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.257+0000 7f6a3f000640 1 --2- 172.21.15.102:0/851212159 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38104320 0x7f6a3810abb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009ca0 tx=0x7f6a3002a8c0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.257+0000 7f6a3f000640 1 -- 172.21.15.102:0/851212159 shutdown_connections 2024-04-26T20:32:56.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.257+0000 7f6a3f000640 1 --2- 172.21.15.102:0/851212159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a38103960 0x7f6a38103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.257+0000 7f6a3f000640 1 --2- 172.21.15.102:0/851212159 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38104320 0x7f6a3810abb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.257+0000 7f6a3f000640 1 --2- 172.21.15.102:0/851212159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38102770 0x7f6a38102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.257+0000 7f6a3f000640 1 -- 172.21.15.102:0/851212159 >> 172.21.15.102:0/851212159 conn(0x7f6a380fdcc0 msgr2=0x7f6a38100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:56.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.257+0000 7f6a3f000640 1 -- 172.21.15.102:0/851212159 shutdown_connections 2024-04-26T20:32:56.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.257+0000 7f6a3f000640 1 -- 172.21.15.102:0/851212159 wait complete. 2024-04-26T20:32:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.258+0000 7f6a3f000640 1 Processor -- start 2024-04-26T20:32:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.258+0000 7f6a3f000640 1 -- start start 2024-04-26T20:32:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.258+0000 7f6a3f000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38102770 0x7f6a3819ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.258+0000 7f6a3f000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a38103960 0x7f6a3819b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.258+0000 7f6a3f000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38104320 0x7f6a381a20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.258+0000 7f6a3f000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3810e2f0 con 0x7f6a38103960 2024-04-26T20:32:56.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.258+0000 7f6a3f000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3810e170 con 0x7f6a38102770 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.258+0000 7f6a3f000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3810e470 con 0x7f6a38104320 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.258+0000 7f6a3dc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38102770 0x7f6a3819ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a3dc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38102770 0x7f6a3819ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:36728/0 (socket says 172.21.15.102:36728) 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a3dc00640 1 -- 172.21.15.102:0/169759326 learned_addr learned my addr 172.21.15.102:0/169759326 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a3d200640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a38103960 0x7f6a3819b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a3e600640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38104320 0x7f6a381a20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a3dc00640 1 -- 172.21.15.102:0/169759326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38104320 msgr2=0x7f6a381a20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a3dc00640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38104320 0x7f6a381a20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a3dc00640 1 -- 172.21.15.102:0/169759326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a38103960 msgr2=0x7f6a3819b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a3dc00640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a38103960 0x7f6a3819b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a3dc00640 1 -- 172.21.15.102:0/169759326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a381a2750 con 0x7f6a38102770 2024-04-26T20:32:56.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a3dc00640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38102770 0x7f6a3819ab30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00e9c0 tx=0x7f6a2c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:56.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.259+0000 7f6a36000640 1 -- 172.21.15.102:0/169759326 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00cd80 con 0x7f6a38102770 2024-04-26T20:32:56.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.260+0000 7f6a36000640 1 -- 172.21.15.102:0/169759326 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a2c0035f0 con 0x7f6a38102770 2024-04-26T20:32:56.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.260+0000 7f6a36000640 1 -- 172.21.15.102:0/169759326 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c010690 con 0x7f6a38102770 2024-04-26T20:32:56.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.260+0000 7f6a3f000640 1 -- 172.21.15.102:0/169759326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a381a2a40 con 0x7f6a38102770 2024-04-26T20:32:56.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.260+0000 7f6a3f000640 1 -- 172.21.15.102:0/169759326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a381a2f40 con 0x7f6a38102770 2024-04-26T20:32:56.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.261+0000 7f6a3f000640 1 -- 172.21.15.102:0/169759326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a381a31d0 con 0x7f6a38102770 2024-04-26T20:32:56.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.264+0000 7f6a36000640 1 -- 172.21.15.102:0/169759326 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f6a2c004000 con 0x7f6a38102770 2024-04-26T20:32:56.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.265+0000 7f6a36000640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a200a3cc0 0x7f6a200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:56.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.265+0000 7f6a36000640 1 -- 172.21.15.102:0/169759326 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6a2c0d0c30 con 0x7f6a38102770 2024-04-26T20:32:56.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.266+0000 7f6a3d200640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a200a3cc0 0x7f6a200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:56.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.266+0000 7f6a3d200640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a200a3cc0 0x7f6a200a6180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6a3819c040 tx=0x7f6a2400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:56.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.269+0000 7f6a36000640 1 -- 172.21.15.102:0/169759326 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6a2c09d270 con 0x7f6a38102770 2024-04-26T20:32:56.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.498+0000 7f6a3f000640 1 -- 172.21.15.102:0/169759326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a381a3440 con 0x7f6a38102770 2024-04-26T20:32:56.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.499+0000 7f6a36000640 1 -- 172.21.15.102:0/169759326 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11954 (secure 0 0 0) 0x7f6a2c09cb70 con 0x7f6a38102770 2024-04-26T20:32:56.500 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:56.501 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:32:25.769708+0000","modified":"2024-04-26T20:32:55.699799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24593,"mds_1":24565,"mds_2":24547,"mds_3":24533,"mds_4":24526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24565,24593,24547,24533,24526]},"id":1}]} 2024-04-26T20:32:56.501 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-26T20:32:56.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 -- 172.21.15.102:0/169759326 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a200a3cc0 msgr2=0x7f6a200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a200a3cc0 0x7f6a200a6180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6a3819c040 tx=0x7f6a2400b410 comp rx=0 tx=0).stop 2024-04-26T20:32:56.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 -- 172.21.15.102:0/169759326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38102770 msgr2=0x7f6a3819ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38102770 0x7f6a3819ab30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00e9c0 tx=0x7f6a2c00ee80 comp rx=0 tx=0).stop 2024-04-26T20:32:56.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 -- 172.21.15.102:0/169759326 shutdown_connections 2024-04-26T20:32:56.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f6a200a3cc0 0x7f6a200a6180 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a38103960 0x7f6a3819b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38104320 0x7f6a381a20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 --2- 172.21.15.102:0/169759326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38102770 0x7f6a3819ab30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 -- 172.21.15.102:0/169759326 >> 172.21.15.102:0/169759326 conn(0x7f6a380fdcc0 msgr2=0x7f6a38074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:56.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 -- 172.21.15.102:0/169759326 shutdown_connections 2024-04-26T20:32:56.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.502+0000 7f6a3f000640 1 -- 172.21.15.102:0/169759326 wait complete. 2024-04-26T20:32:56.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:32:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.709+0000 7faeda000640 1 -- 172.21.15.102:0/3025789878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faed41049b0 msgr2=0x7faed4104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.709+0000 7faeda000640 1 --2- 172.21.15.102:0/3025789878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faed41049b0 0x7faed4104dd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faec8009900 tx=0x7faec802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.709+0000 7faeda000640 1 -- 172.21.15.102:0/3025789878 shutdown_connections 2024-04-26T20:32:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.709+0000 7faeda000640 1 --2- 172.21.15.102:0/3025789878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faed4106560 0x7faed410cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.709+0000 7faeda000640 1 --2- 172.21.15.102:0/3025789878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faed4105ba0 0x7faed4106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.709+0000 7faeda000640 1 --2- 172.21.15.102:0/3025789878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faed41049b0 0x7faed4104dd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.709+0000 7faeda000640 1 -- 172.21.15.102:0/3025789878 >> 172.21.15.102:0/3025789878 conn(0x7faed40fff40 msgr2=0x7faed4102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.709+0000 7faeda000640 1 -- 172.21.15.102:0/3025789878 shutdown_connections 2024-04-26T20:32:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.709+0000 7faeda000640 1 -- 172.21.15.102:0/3025789878 wait complete. 2024-04-26T20:32:56.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.710+0000 7faeda000640 1 Processor -- start 2024-04-26T20:32:56.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.710+0000 7faeda000640 1 -- start start 2024-04-26T20:32:56.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.710+0000 7faeda000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faed41049b0 0x7faed419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:56.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.710+0000 7faeda000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faed4105ba0 0x7faed419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:56.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.710+0000 7faeda000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faed4106560 0x7faed41a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:56.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.710+0000 7faeda000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faed4110520 con 0x7faed4105ba0 2024-04-26T20:32:56.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.710+0000 7faeda000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faed41103a0 con 0x7faed41049b0 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.710+0000 7faeda000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7faed41106a0 con 0x7faed4106560 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faed4105ba0 0x7faed419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faed4105ba0 0x7faed419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33116/0 (socket says 172.21.15.102:33116) 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed3e00640 1 -- 172.21.15.102:0/2945108522 learned_addr learned my addr 172.21.15.102:0/2945108522 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed9600640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faed4106560 0x7faed41a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed8c00640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faed41049b0 0x7faed419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed3e00640 1 -- 172.21.15.102:0/2945108522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faed4106560 msgr2=0x7faed41a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed3e00640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faed4106560 0x7faed41a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed3e00640 1 -- 172.21.15.102:0/2945108522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faed41049b0 msgr2=0x7faed419ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed3e00640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faed41049b0 0x7faed419ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed3e00640 1 -- 172.21.15.102:0/2945108522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed41a27b0 con 0x7faed4105ba0 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.711+0000 7faed3e00640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faed4105ba0 0x7faed419b0d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faec000e9f0 tx=0x7faec000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:56.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.712+0000 7faed1600640 1 -- 172.21.15.102:0/2945108522 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec000cd80 con 0x7faed4105ba0 2024-04-26T20:32:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.712+0000 7faeda000640 1 -- 172.21.15.102:0/2945108522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed41a2aa0 con 0x7faed4105ba0 2024-04-26T20:32:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.712+0000 7faed1600640 1 -- 172.21.15.102:0/2945108522 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec00054d0 con 0x7faed4105ba0 2024-04-26T20:32:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.712+0000 7faed1600640 1 -- 172.21.15.102:0/2945108522 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0018630 con 0x7faed4105ba0 2024-04-26T20:32:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.712+0000 7faeda000640 1 -- 172.21.15.102:0/2945108522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed41a2fa0 con 0x7faed4105ba0 2024-04-26T20:32:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.713+0000 7faeda000640 1 -- 172.21.15.102:0/2945108522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed410bcd0 con 0x7faed4105ba0 2024-04-26T20:32:56.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.715+0000 7faed1600640 1 -- 172.21.15.102:0/2945108522 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7faec0004e70 con 0x7faed4105ba0 2024-04-26T20:32:56.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.716+0000 7faed1600640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faebc0a3c40 0x7faebc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:56.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.716+0000 7faed1600640 1 -- 172.21.15.102:0/2945108522 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7faec00d0d50 con 0x7faed4105ba0 2024-04-26T20:32:56.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.717+0000 7faed8c00640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faebc0a3c40 0x7faebc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:56.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.717+0000 7faed8c00640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faebc0a3c40 0x7faebc0a6100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faec8005ab0 tx=0x7faec8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:56.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.721+0000 7faed1600640 1 -- 172.21.15.102:0/2945108522 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faec009d330 con 0x7faed4105ba0 2024-04-26T20:32:56.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:56 smithi102 ceph-mon[28094]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:56.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:56 smithi102 ceph-mon[28094]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:56.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:56 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] up:active 2024-04-26T20:32:56.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:56 smithi102 ceph-mon[28094]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-26T20:32:56.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:56 smithi102 ceph-mon[28094]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T20:32:56.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:56 smithi102 ceph-mon[28094]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T20:32:56.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:56 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/169759326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.924+0000 7faeda000640 1 -- 172.21.15.102:0/2945108522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faed402dca0 con 0x7faed4105ba0 2024-04-26T20:32:56.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.928+0000 7faed1600640 1 -- 172.21.15.102:0/2945108522 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24164 (secure 0 0 0) 0x7faec009cc30 con 0x7faed4105ba0 2024-04-26T20:32:56.930 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:56.930 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:32:30.827511+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 -- 172.21.15.102:0/2945108522 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faebc0a3c40 msgr2=0x7faebc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faebc0a3c40 0x7faebc0a6100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faec8005ab0 tx=0x7faec8005a20 comp rx=0 tx=0).stop 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 -- 172.21.15.102:0/2945108522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faed4105ba0 msgr2=0x7faed419b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faed4105ba0 0x7faed419b0d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faec000e9f0 tx=0x7faec000edf0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 -- 172.21.15.102:0/2945108522 shutdown_connections 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faebc0a3c40 0x7faebc0a6100 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faed4105ba0 0x7faed419b0d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faed4106560 0x7faed41a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 --2- 172.21.15.102:0/2945108522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faed41049b0 0x7faed419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 -- 172.21.15.102:0/2945108522 >> 172.21.15.102:0/2945108522 conn(0x7faed40fff40 msgr2=0x7faed4108ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 -- 172.21.15.102:0/2945108522 shutdown_connections 2024-04-26T20:32:56.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:56.931+0000 7faeda000640 1 -- 172.21.15.102:0/2945108522 wait complete. 2024-04-26T20:32:56.950 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T20:32:56.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T20:32:57.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:56 smithi112 ceph-mon[32769]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:57.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:56 smithi112 ceph-mon[32769]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:57.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:56 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] up:active 2024-04-26T20:32:57.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:56 smithi112 ceph-mon[32769]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-26T20:32:57.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:56 smithi112 ceph-mon[32769]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T20:32:57.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:56 smithi112 ceph-mon[32769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T20:32:57.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:56 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/169759326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:57.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:56 smithi130 ceph-mon[32704]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:57.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:56 smithi130 ceph-mon[32704]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:57.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:56 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] up:active 2024-04-26T20:32:57.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:56 smithi130 ceph-mon[32704]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-26T20:32:57.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:56 smithi130 ceph-mon[32704]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T20:32:57.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:56 smithi130 ceph-mon[32704]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T20:32:57.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:56 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/169759326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:32:57.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.149+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1183719161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4e0104990 msgr2=0x7fa4e0104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.149+0000 7fa4e5e00640 1 --2- 172.21.15.102:0/1183719161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4e0104990 0x7fa4e0104db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0009900 tx=0x7fa4d002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.149+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1183719161 shutdown_connections 2024-04-26T20:32:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.149+0000 7fa4e5e00640 1 --2- 172.21.15.102:0/1183719161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4e0106540 0x7fa4e010cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.149+0000 7fa4e5e00640 1 --2- 172.21.15.102:0/1183719161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e0105b80 0x7fa4e0106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.149+0000 7fa4e5e00640 1 --2- 172.21.15.102:0/1183719161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4e0104990 0x7fa4e0104db0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.149+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1183719161 >> 172.21.15.102:0/1183719161 conn(0x7fa4e00fff00 msgr2=0x7fa4e0102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.149+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1183719161 shutdown_connections 2024-04-26T20:32:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.149+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1183719161 wait complete. 2024-04-26T20:32:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.150+0000 7fa4e5e00640 1 Processor -- start 2024-04-26T20:32:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.150+0000 7fa4e5e00640 1 -- start start 2024-04-26T20:32:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4e0104990 0x7fa4e019ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4e0105b80 0x7fa4e019b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e0106540 0x7fa4e01a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e01104c0 con 0x7fa4e0105b80 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e0110340 con 0x7fa4e0106540 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5e00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e0110640 con 0x7fa4e0104990 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4d7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4e0105b80 0x7fa4e019b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e4a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4e0104990 0x7fa4e019ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e0106540 0x7fa4e01a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e4a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4e0104990 0x7fa4e019ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:53184/0 (socket says 172.21.15.102:53184) 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e0106540 0x7fa4e01a2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:36782/0 (socket says 172.21.15.102:36782) 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e4a00640 1 -- 172.21.15.102:0/1555166581 learned_addr learned my addr 172.21.15.102:0/1555166581 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5400640 1 -- 172.21.15.102:0/1555166581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4e0104990 msgr2=0x7fa4e019ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5400640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4e0104990 0x7fa4e019ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5400640 1 -- 172.21.15.102:0/1555166581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4e0105b80 msgr2=0x7fa4e019b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5400640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4e0105b80 0x7fa4e019b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4e5400640 1 -- 172.21.15.102:0/1555166581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e01a27b0 con 0x7fa4e0106540 2024-04-26T20:32:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.151+0000 7fa4d7e00640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4e0105b80 0x7fa4e019b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:32:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.152+0000 7fa4e5400640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e0106540 0x7fa4e01a2140 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009fd0 tx=0x7fa4d800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.152+0000 7fa4de000640 1 -- 172.21.15.102:0/1555166581 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d800ce20 con 0x7fa4e0106540 2024-04-26T20:32:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.152+0000 7fa4de000640 1 -- 172.21.15.102:0/1555166581 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4d8005020 con 0x7fa4e0106540 2024-04-26T20:32:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.152+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1555166581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e01a2aa0 con 0x7fa4e0106540 2024-04-26T20:32:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.152+0000 7fa4de000640 1 -- 172.21.15.102:0/1555166581 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d8010690 con 0x7fa4e0106540 2024-04-26T20:32:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.152+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1555166581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e01a2ff0 con 0x7fa4e0106540 2024-04-26T20:32:57.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.153+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1555166581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4e010bdd0 con 0x7fa4e0106540 2024-04-26T20:32:57.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.155+0000 7fa4de000640 1 -- 172.21.15.102:0/1555166581 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa4d80054d0 con 0x7fa4e0106540 2024-04-26T20:32:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.156+0000 7fa4de000640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4c40a3c40 0x7fa4c40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.156+0000 7fa4de000640 1 -- 172.21.15.102:0/1555166581 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa4d80d0f00 con 0x7fa4e0106540 2024-04-26T20:32:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.156+0000 7fa4e4a00640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4c40a3c40 0x7fa4c40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:57.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.157+0000 7fa4e4a00640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4c40a3c40 0x7fa4c40a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0000c00 tx=0x7fa4d00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.160+0000 7fa4de000640 1 -- 172.21.15.102:0/1555166581 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa4d809d4e0 con 0x7fa4e0106540 2024-04-26T20:32:57.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.362+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1555166581 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa4e0062d70 con 0x7fa4c40a3c40 2024-04-26T20:32:57.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.398+0000 7fa4de000640 1 -- 172.21.15.102:0/1555166581 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340333 (secure 0 0 0) 0x7fa4e0062d70 con 0x7fa4c40a3c40 2024-04-26T20:32:57.404 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:57.406 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-26T20:32:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.405+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1555166581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4c40a3c40 msgr2=0x7fa4c40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.405+0000 7fa4e5e00640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4c40a3c40 0x7fa4c40a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0000c00 tx=0x7fa4d00314a0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.405+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1555166581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e0106540 msgr2=0x7fa4e01a2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.406+0000 7fa4e5e00640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e0106540 0x7fa4e01a2140 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009fd0 tx=0x7fa4d800ef10 comp rx=0 tx=0).stop 2024-04-26T20:32:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.406+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1555166581 shutdown_connections 2024-04-26T20:32:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.406+0000 7fa4e5e00640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4c40a3c40 0x7fa4c40a6100 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.406+0000 7fa4e5e00640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4e0105b80 0x7fa4e019b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.406+0000 7fa4e5e00640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e0106540 0x7fa4e01a2140 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.406+0000 7fa4e5e00640 1 --2- 172.21.15.102:0/1555166581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4e0104990 0x7fa4e019ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.406+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1555166581 >> 172.21.15.102:0/1555166581 conn(0x7fa4e00fff00 msgr2=0x7fa4e0108fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:57.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.406+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1555166581 shutdown_connections 2024-04-26T20:32:57.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.406+0000 7fa4e5e00640 1 -- 172.21.15.102:0/1555166581 wait complete. 2024-04-26T20:32:57.427 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-04-26T20:32:56.525413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598694,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":340,"num_write_kb":2993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":350144,"kb_used_data":13472,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124591680,"statfs":{"total":1151940427776,"available":1151581880320,"internally_reserved":0,"allocated":13795328,"data_stored":7521062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":344634682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5467,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005876"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551581+0000","last_change":"2024-04-26T20:32:26.249504+0000","last_active":"2024-04-26T20:32:31.551581+0000","last_peered":"2024-04-26T20:32:31.551581+0000","last_clean":"2024-04-26T20:32:31.551581+0000","last_became_active":"2024-04-26T20:32:26.249188+0000","last_became_peered":"2024-04-26T20:32:26.249188+0000","last_unstale":"2024-04-26T20:32:31.551581+0000","last_undegraded":"2024-04-26T20:32:31.551581+0000","last_fullsized":"2024-04-26T20:32:31.551581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:28:15.906199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.367854+0000","last_change":"2024-04-26T20:32:26.246991+0000","last_active":"2024-04-26T20:32:31.367854+0000","last_peered":"2024-04-26T20:32:31.367854+0000","last_clean":"2024-04-26T20:32:31.367854+0000","last_became_active":"2024-04-26T20:32:26.245552+0000","last_became_peered":"2024-04-26T20:32:26.245552+0000","last_unstale":"2024-04-26T20:32:31.367854+0000","last_undegraded":"2024-04-26T20:32:31.367854+0000","last_fullsized":"2024-04-26T20:32:31.367854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:59:46.487659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366359+0000","last_change":"2024-04-26T20:32:26.241451+0000","last_active":"2024-04-26T20:32:31.366359+0000","last_peered":"2024-04-26T20:32:31.366359+0000","last_clean":"2024-04-26T20:32:31.366359+0000","last_became_active":"2024-04-26T20:32:26.240012+0000","last_became_peered":"2024-04-26T20:32:26.240012+0000","last_unstale":"2024-04-26T20:32:31.366359+0000","last_undegraded":"2024-04-26T20:32:31.366359+0000","last_fullsized":"2024-04-26T20:32:31.366359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:43:12.776082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550461+0000","last_change":"2024-04-26T20:32:26.224138+0000","last_active":"2024-04-26T20:32:31.550461+0000","last_peered":"2024-04-26T20:32:31.550461+0000","last_clean":"2024-04-26T20:32:31.550461+0000","last_became_active":"2024-04-26T20:32:26.223827+0000","last_became_peered":"2024-04-26T20:32:26.223827+0000","last_unstale":"2024-04-26T20:32:31.550461+0000","last_undegraded":"2024-04-26T20:32:31.550461+0000","last_fullsized":"2024-04-26T20:32:31.550461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:38:33.323001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548862+0000","last_change":"2024-04-26T20:32:26.240298+0000","last_active":"2024-04-26T20:32:31.548862+0000","last_peered":"2024-04-26T20:32:31.548862+0000","last_clean":"2024-04-26T20:32:31.548862+0000","last_became_active":"2024-04-26T20:32:26.239933+0000","last_became_peered":"2024-04-26T20:32:26.239933+0000","last_unstale":"2024-04-26T20:32:31.548862+0000","last_undegraded":"2024-04-26T20:32:31.548862+0000","last_fullsized":"2024-04-26T20:32:31.548862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:00.618136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550850+0000","last_change":"2024-04-26T20:32:26.233053+0000","last_active":"2024-04-26T20:32:31.550850+0000","last_peered":"2024-04-26T20:32:31.550850+0000","last_clean":"2024-04-26T20:32:31.550850+0000","last_became_active":"2024-04-26T20:32:26.232727+0000","last_became_peered":"2024-04-26T20:32:26.232727+0000","last_unstale":"2024-04-26T20:32:31.550850+0000","last_undegraded":"2024-04-26T20:32:31.550850+0000","last_fullsized":"2024-04-26T20:32:31.550850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:35:41.120741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.360730+0000","last_change":"2024-04-26T20:32:26.232600+0000","last_active":"2024-04-26T20:32:31.360730+0000","last_peered":"2024-04-26T20:32:31.360730+0000","last_clean":"2024-04-26T20:32:31.360730+0000","last_became_active":"2024-04-26T20:32:26.232320+0000","last_became_peered":"2024-04-26T20:32:26.232320+0000","last_unstale":"2024-04-26T20:32:31.360730+0000","last_undegraded":"2024-04-26T20:32:31.360730+0000","last_fullsized":"2024-04-26T20:32:31.360730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:18.199940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.362800+0000","last_change":"2024-04-26T20:32:26.215513+0000","last_active":"2024-04-26T20:32:31.362800+0000","last_peered":"2024-04-26T20:32:31.362800+0000","last_clean":"2024-04-26T20:32:31.362800+0000","last_became_active":"2024-04-26T20:32:26.215110+0000","last_became_peered":"2024-04-26T20:32:26.215110+0000","last_unstale":"2024-04-26T20:32:31.362800+0000","last_undegraded":"2024-04-26T20:32:31.362800+0000","last_fullsized":"2024-04-26T20:32:31.362800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:11.356841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548695+0000","last_change":"2024-04-26T20:32:26.230224+0000","last_active":"2024-04-26T20:32:31.548695+0000","last_peered":"2024-04-26T20:32:31.548695+0000","last_clean":"2024-04-26T20:32:31.548695+0000","last_became_active":"2024-04-26T20:32:26.229865+0000","last_became_peered":"2024-04-26T20:32:26.229865+0000","last_unstale":"2024-04-26T20:32:31.548695+0000","last_undegraded":"2024-04-26T20:32:31.548695+0000","last_fullsized":"2024-04-26T20:32:31.548695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:03.060925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551132+0000","last_change":"2024-04-26T20:32:26.219841+0000","last_active":"2024-04-26T20:32:31.551132+0000","last_peered":"2024-04-26T20:32:31.551132+0000","last_clean":"2024-04-26T20:32:31.551132+0000","last_became_active":"2024-04-26T20:32:26.219505+0000","last_became_peered":"2024-04-26T20:32:26.219505+0000","last_unstale":"2024-04-26T20:32:31.551132+0000","last_undegraded":"2024-04-26T20:32:31.551132+0000","last_fullsized":"2024-04-26T20:32:31.551132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:11.514510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361472+0000","last_change":"2024-04-26T20:32:26.226427+0000","last_active":"2024-04-26T20:32:31.361472+0000","last_peered":"2024-04-26T20:32:31.361472+0000","last_clean":"2024-04-26T20:32:31.361472+0000","last_became_active":"2024-04-26T20:32:26.226112+0000","last_became_peered":"2024-04-26T20:32:26.226112+0000","last_unstale":"2024-04-26T20:32:31.361472+0000","last_undegraded":"2024-04-26T20:32:31.361472+0000","last_fullsized":"2024-04-26T20:32:31.361472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:56.506369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550356+0000","last_change":"2024-04-26T20:32:26.222756+0000","last_active":"2024-04-26T20:32:31.550356+0000","last_peered":"2024-04-26T20:32:31.550356+0000","last_clean":"2024-04-26T20:32:31.550356+0000","last_became_active":"2024-04-26T20:32:26.222342+0000","last_became_peered":"2024-04-26T20:32:26.222342+0000","last_unstale":"2024-04-26T20:32:31.550356+0000","last_undegraded":"2024-04-26T20:32:31.550356+0000","last_fullsized":"2024-04-26T20:32:31.550356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:11.543434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.552125+0000","last_change":"2024-04-26T20:32:26.242864+0000","last_active":"2024-04-26T20:32:31.552125+0000","last_peered":"2024-04-26T20:32:31.552125+0000","last_clean":"2024-04-26T20:32:31.552125+0000","last_became_active":"2024-04-26T20:32:26.242511+0000","last_became_peered":"2024-04-26T20:32:26.242511+0000","last_unstale":"2024-04-26T20:32:31.552125+0000","last_undegraded":"2024-04-26T20:32:31.552125+0000","last_fullsized":"2024-04-26T20:32:31.552125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:26:37.412901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.365379+0000","last_change":"2024-04-26T20:32:26.247783+0000","last_active":"2024-04-26T20:32:31.365379+0000","last_peered":"2024-04-26T20:32:31.365379+0000","last_clean":"2024-04-26T20:32:31.365379+0000","last_became_active":"2024-04-26T20:32:26.247466+0000","last_became_peered":"2024-04-26T20:32:26.247466+0000","last_unstale":"2024-04-26T20:32:31.365379+0000","last_undegraded":"2024-04-26T20:32:31.365379+0000","last_fullsized":"2024-04-26T20:32:31.365379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:36:59.424377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.364729+0000","last_change":"2024-04-26T20:32:26.249823+0000","last_active":"2024-04-26T20:32:31.364729+0000","last_peered":"2024-04-26T20:32:31.364729+0000","last_clean":"2024-04-26T20:32:31.364729+0000","last_became_active":"2024-04-26T20:32:26.248954+0000","last_became_peered":"2024-04-26T20:32:26.248954+0000","last_unstale":"2024-04-26T20:32:31.364729+0000","last_undegraded":"2024-04-26T20:32:31.364729+0000","last_fullsized":"2024-04-26T20:32:31.364729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:55:44.246537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548108+0000","last_change":"2024-04-26T20:32:26.241997+0000","last_active":"2024-04-26T20:32:31.548108+0000","last_peered":"2024-04-26T20:32:31.548108+0000","last_clean":"2024-04-26T20:32:31.548108+0000","last_became_active":"2024-04-26T20:32:26.240856+0000","last_became_peered":"2024-04-26T20:32:26.240856+0000","last_unstale":"2024-04-26T20:32:31.548108+0000","last_undegraded":"2024-04-26T20:32:31.548108+0000","last_fullsized":"2024-04-26T20:32:31.548108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:53.895994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551455+0000","last_change":"2024-04-26T20:32:26.241097+0000","last_active":"2024-04-26T20:32:31.551455+0000","last_peered":"2024-04-26T20:32:31.551455+0000","last_clean":"2024-04-26T20:32:31.551455+0000","last_became_active":"2024-04-26T20:32:26.240770+0000","last_became_peered":"2024-04-26T20:32:26.240770+0000","last_unstale":"2024-04-26T20:32:31.551455+0000","last_undegraded":"2024-04-26T20:32:31.551455+0000","last_fullsized":"2024-04-26T20:32:31.551455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:10.443314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551349+0000","last_change":"2024-04-26T20:32:26.232279+0000","last_active":"2024-04-26T20:32:31.551349+0000","last_peered":"2024-04-26T20:32:31.551349+0000","last_clean":"2024-04-26T20:32:31.551349+0000","last_became_active":"2024-04-26T20:32:26.231256+0000","last_became_peered":"2024-04-26T20:32:26.231256+0000","last_unstale":"2024-04-26T20:32:31.551349+0000","last_undegraded":"2024-04-26T20:32:31.551349+0000","last_fullsized":"2024-04-26T20:32:31.551349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:26.484932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550404+0000","last_change":"2024-04-26T20:32:26.249876+0000","last_active":"2024-04-26T20:32:31.550404+0000","last_peered":"2024-04-26T20:32:31.550404+0000","last_clean":"2024-04-26T20:32:31.550404+0000","last_became_active":"2024-04-26T20:32:26.249416+0000","last_became_peered":"2024-04-26T20:32:26.249416+0000","last_unstale":"2024-04-26T20:32:31.550404+0000","last_undegraded":"2024-04-26T20:32:31.550404+0000","last_fullsized":"2024-04-26T20:32:31.550404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:54.616941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549613+0000","last_change":"2024-04-26T20:32:30.389128+0000","last_active":"2024-04-26T20:32:31.549613+0000","last_peered":"2024-04-26T20:32:31.549613+0000","last_clean":"2024-04-26T20:32:31.549613+0000","last_became_active":"2024-04-26T20:32:30.388668+0000","last_became_peered":"2024-04-26T20:32:30.388668+0000","last_unstale":"2024-04-26T20:32:31.549613+0000","last_undegraded":"2024-04-26T20:32:31.549613+0000","last_fullsized":"2024-04-26T20:32:31.549613+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:12.019691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:54.687894+0000","last_change":"2024-04-26T20:32:25.278417+0000","last_active":"2024-04-26T20:32:54.687894+0000","last_peered":"2024-04-26T20:32:54.687894+0000","last_clean":"2024-04-26T20:32:54.687894+0000","last_became_active":"2024-04-26T20:32:25.278004+0000","last_became_peered":"2024-04-26T20:32:25.278004+0000","last_unstale":"2024-04-26T20:32:54.687894+0000","last_undegraded":"2024-04-26T20:32:54.687894+0000","last_fullsized":"2024-04-26T20:32:54.687894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:51.458285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.365337+0000","last_change":"2024-04-26T20:32:26.229015+0000","last_active":"2024-04-26T20:32:31.365337+0000","last_peered":"2024-04-26T20:32:31.365337+0000","last_clean":"2024-04-26T20:32:31.365337+0000","last_became_active":"2024-04-26T20:32:26.228729+0000","last_became_peered":"2024-04-26T20:32:26.228729+0000","last_unstale":"2024-04-26T20:32:31.365337+0000","last_undegraded":"2024-04-26T20:32:31.365337+0000","last_fullsized":"2024-04-26T20:32:31.365337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:20.014133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549341+0000","last_change":"2024-04-26T20:32:30.383196+0000","last_active":"2024-04-26T20:32:31.549341+0000","last_peered":"2024-04-26T20:32:31.549341+0000","last_clean":"2024-04-26T20:32:31.549341+0000","last_became_active":"2024-04-26T20:32:30.382831+0000","last_became_peered":"2024-04-26T20:32:30.382831+0000","last_unstale":"2024-04-26T20:32:31.549341+0000","last_undegraded":"2024-04-26T20:32:31.549341+0000","last_fullsized":"2024-04-26T20:32:31.549341+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:48:13.696394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366869+0000","last_change":"2024-04-26T20:32:25.294227+0000","last_active":"2024-04-26T20:32:31.366869+0000","last_peered":"2024-04-26T20:32:31.366869+0000","last_clean":"2024-04-26T20:32:31.366869+0000","last_became_active":"2024-04-26T20:32:25.287681+0000","last_became_peered":"2024-04-26T20:32:25.287681+0000","last_unstale":"2024-04-26T20:32:31.366869+0000","last_undegraded":"2024-04-26T20:32:31.366869+0000","last_fullsized":"2024-04-26T20:32:31.366869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:25.567135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.368440+0000","last_change":"2024-04-26T20:32:26.251364+0000","last_active":"2024-04-26T20:32:31.368440+0000","last_peered":"2024-04-26T20:32:31.368440+0000","last_clean":"2024-04-26T20:32:31.368440+0000","last_became_active":"2024-04-26T20:32:26.251037+0000","last_became_peered":"2024-04-26T20:32:26.251037+0000","last_unstale":"2024-04-26T20:32:31.368440+0000","last_undegraded":"2024-04-26T20:32:31.368440+0000","last_fullsized":"2024-04-26T20:32:31.368440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:17:41.813889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550321+0000","last_change":"2024-04-26T20:32:30.385134+0000","last_active":"2024-04-26T20:32:31.550321+0000","last_peered":"2024-04-26T20:32:31.550321+0000","last_clean":"2024-04-26T20:32:31.550321+0000","last_became_active":"2024-04-26T20:32:30.384694+0000","last_became_peered":"2024-04-26T20:32:30.384694+0000","last_unstale":"2024-04-26T20:32:31.550321+0000","last_undegraded":"2024-04-26T20:32:31.550321+0000","last_fullsized":"2024-04-26T20:32:31.550321+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:59.622623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'11","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:54.686164+0000","last_change":"2024-04-26T20:32:25.267156+0000","last_active":"2024-04-26T20:32:54.686164+0000","last_peered":"2024-04-26T20:32:54.686164+0000","last_clean":"2024-04-26T20:32:54.686164+0000","last_became_active":"2024-04-26T20:32:25.266113+0000","last_became_peered":"2024-04-26T20:32:25.266113+0000","last_unstale":"2024-04-26T20:32:54.686164+0000","last_undegraded":"2024-04-26T20:32:54.686164+0000","last_fullsized":"2024-04-26T20:32:54.686164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:30:38.798437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548869+0000","last_change":"2024-04-26T20:32:26.249908+0000","last_active":"2024-04-26T20:32:31.548869+0000","last_peered":"2024-04-26T20:32:31.548869+0000","last_clean":"2024-04-26T20:32:31.548869+0000","last_became_active":"2024-04-26T20:32:26.249576+0000","last_became_peered":"2024-04-26T20:32:26.249576+0000","last_unstale":"2024-04-26T20:32:31.548869+0000","last_undegraded":"2024-04-26T20:32:31.548869+0000","last_fullsized":"2024-04-26T20:32:31.548869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:37:09.611624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549802+0000","last_change":"2024-04-26T20:32:30.391456+0000","last_active":"2024-04-26T20:32:31.549802+0000","last_peered":"2024-04-26T20:32:31.549802+0000","last_clean":"2024-04-26T20:32:31.549802+0000","last_became_active":"2024-04-26T20:32:30.391028+0000","last_became_peered":"2024-04-26T20:32:30.391028+0000","last_unstale":"2024-04-26T20:32:31.549802+0000","last_undegraded":"2024-04-26T20:32:31.549802+0000","last_fullsized":"2024-04-26T20:32:31.549802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:55:17.987421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:54.685217+0000","last_change":"2024-04-26T20:32:25.264519+0000","last_active":"2024-04-26T20:32:54.685217+0000","last_peered":"2024-04-26T20:32:54.685217+0000","last_clean":"2024-04-26T20:32:54.685217+0000","last_became_active":"2024-04-26T20:32:25.264189+0000","last_became_peered":"2024-04-26T20:32:25.264189+0000","last_unstale":"2024-04-26T20:32:54.685217+0000","last_undegraded":"2024-04-26T20:32:54.685217+0000","last_fullsized":"2024-04-26T20:32:54.685217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:12:56.167119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551106+0000","last_change":"2024-04-26T20:32:26.226281+0000","last_active":"2024-04-26T20:32:31.551106+0000","last_peered":"2024-04-26T20:32:31.551106+0000","last_clean":"2024-04-26T20:32:31.551106+0000","last_became_active":"2024-04-26T20:32:26.225958+0000","last_became_peered":"2024-04-26T20:32:26.225958+0000","last_unstale":"2024-04-26T20:32:31.551106+0000","last_undegraded":"2024-04-26T20:32:31.551106+0000","last_fullsized":"2024-04-26T20:32:31.551106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:22:30.914293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550306+0000","last_change":"2024-04-26T20:32:30.379734+0000","last_active":"2024-04-26T20:32:31.550306+0000","last_peered":"2024-04-26T20:32:31.550306+0000","last_clean":"2024-04-26T20:32:31.550306+0000","last_became_active":"2024-04-26T20:32:30.379240+0000","last_became_peered":"2024-04-26T20:32:30.379240+0000","last_unstale":"2024-04-26T20:32:31.550306+0000","last_undegraded":"2024-04-26T20:32:31.550306+0000","last_fullsized":"2024-04-26T20:32:31.550306+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:05:46.669583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.657470+0000","last_change":"2024-04-26T20:32:25.268389+0000","last_active":"2024-04-26T20:32:51.657470+0000","last_peered":"2024-04-26T20:32:51.657470+0000","last_clean":"2024-04-26T20:32:51.657470+0000","last_became_active":"2024-04-26T20:32:25.268060+0000","last_became_peered":"2024-04-26T20:32:25.268060+0000","last_unstale":"2024-04-26T20:32:51.657470+0000","last_undegraded":"2024-04-26T20:32:51.657470+0000","last_fullsized":"2024-04-26T20:32:51.657470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:38.636609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548835+0000","last_change":"2024-04-26T20:32:26.235743+0000","last_active":"2024-04-26T20:32:31.548835+0000","last_peered":"2024-04-26T20:32:31.548835+0000","last_clean":"2024-04-26T20:32:31.548835+0000","last_became_active":"2024-04-26T20:32:26.235375+0000","last_became_peered":"2024-04-26T20:32:26.235375+0000","last_unstale":"2024-04-26T20:32:31.548835+0000","last_undegraded":"2024-04-26T20:32:31.548835+0000","last_fullsized":"2024-04-26T20:32:31.548835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:27.673837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550310+0000","last_change":"2024-04-26T20:32:30.390260+0000","last_active":"2024-04-26T20:32:31.550310+0000","last_peered":"2024-04-26T20:32:31.550310+0000","last_clean":"2024-04-26T20:32:31.550310+0000","last_became_active":"2024-04-26T20:32:30.389796+0000","last_became_peered":"2024-04-26T20:32:30.389796+0000","last_unstale":"2024-04-26T20:32:31.550310+0000","last_undegraded":"2024-04-26T20:32:31.550310+0000","last_fullsized":"2024-04-26T20:32:31.550310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:16:13.399131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:52.667686+0000","last_change":"2024-04-26T20:32:25.301063+0000","last_active":"2024-04-26T20:32:52.667686+0000","last_peered":"2024-04-26T20:32:52.667686+0000","last_clean":"2024-04-26T20:32:52.667686+0000","last_became_active":"2024-04-26T20:32:25.300755+0000","last_became_peered":"2024-04-26T20:32:25.300755+0000","last_unstale":"2024-04-26T20:32:52.667686+0000","last_undegraded":"2024-04-26T20:32:52.667686+0000","last_fullsized":"2024-04-26T20:32:52.667686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:50.542961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550467+0000","last_change":"2024-04-26T20:32:26.223507+0000","last_active":"2024-04-26T20:32:31.550467+0000","last_peered":"2024-04-26T20:32:31.550467+0000","last_clean":"2024-04-26T20:32:31.550467+0000","last_became_active":"2024-04-26T20:32:26.222991+0000","last_became_peered":"2024-04-26T20:32:26.222991+0000","last_unstale":"2024-04-26T20:32:31.550467+0000","last_undegraded":"2024-04-26T20:32:31.550467+0000","last_fullsized":"2024-04-26T20:32:31.550467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:53:25.207884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551519+0000","last_change":"2024-04-26T20:32:30.389354+0000","last_active":"2024-04-26T20:32:31.551519+0000","last_peered":"2024-04-26T20:32:31.551519+0000","last_clean":"2024-04-26T20:32:31.551519+0000","last_became_active":"2024-04-26T20:32:30.388848+0000","last_became_peered":"2024-04-26T20:32:30.388848+0000","last_unstale":"2024-04-26T20:32:31.551519+0000","last_undegraded":"2024-04-26T20:32:31.551519+0000","last_fullsized":"2024-04-26T20:32:31.551519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:12.615750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:52.669205+0000","last_change":"2024-04-26T20:32:25.292694+0000","last_active":"2024-04-26T20:32:52.669205+0000","last_peered":"2024-04-26T20:32:52.669205+0000","last_clean":"2024-04-26T20:32:52.669205+0000","last_became_active":"2024-04-26T20:32:25.283757+0000","last_became_peered":"2024-04-26T20:32:25.283757+0000","last_unstale":"2024-04-26T20:32:52.669205+0000","last_undegraded":"2024-04-26T20:32:52.669205+0000","last_fullsized":"2024-04-26T20:32:52.669205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:05:15.294842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.363566+0000","last_change":"2024-04-26T20:32:26.232770+0000","last_active":"2024-04-26T20:32:31.363566+0000","last_peered":"2024-04-26T20:32:31.363566+0000","last_clean":"2024-04-26T20:32:31.363566+0000","last_became_active":"2024-04-26T20:32:26.232381+0000","last_became_peered":"2024-04-26T20:32:26.232381+0000","last_unstale":"2024-04-26T20:32:31.363566+0000","last_undegraded":"2024-04-26T20:32:31.363566+0000","last_fullsized":"2024-04-26T20:32:31.363566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:16:43.310582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366947+0000","last_change":"2024-04-26T20:32:30.389690+0000","last_active":"2024-04-26T20:32:31.366947+0000","last_peered":"2024-04-26T20:32:31.366947+0000","last_clean":"2024-04-26T20:32:31.366947+0000","last_became_active":"2024-04-26T20:32:30.389258+0000","last_became_peered":"2024-04-26T20:32:30.389258+0000","last_unstale":"2024-04-26T20:32:31.366947+0000","last_undegraded":"2024-04-26T20:32:31.366947+0000","last_fullsized":"2024-04-26T20:32:31.366947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:36.528757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.656122+0000","last_change":"2024-04-26T20:32:25.267681+0000","last_active":"2024-04-26T20:32:51.656122+0000","last_peered":"2024-04-26T20:32:51.656122+0000","last_clean":"2024-04-26T20:32:51.656122+0000","last_became_active":"2024-04-26T20:32:25.267351+0000","last_became_peered":"2024-04-26T20:32:25.267351+0000","last_unstale":"2024-04-26T20:32:51.656122+0000","last_undegraded":"2024-04-26T20:32:51.656122+0000","last_fullsized":"2024-04-26T20:32:51.656122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:00:26.339458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550206+0000","last_change":"2024-04-26T20:32:26.243638+0000","last_active":"2024-04-26T20:32:31.550206+0000","last_peered":"2024-04-26T20:32:31.550206+0000","last_clean":"2024-04-26T20:32:31.550206+0000","last_became_active":"2024-04-26T20:32:26.241783+0000","last_became_peered":"2024-04-26T20:32:26.241783+0000","last_unstale":"2024-04-26T20:32:31.550206+0000","last_undegraded":"2024-04-26T20:32:31.550206+0000","last_fullsized":"2024-04-26T20:32:31.550206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:20.821967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.364193+0000","last_change":"2024-04-26T20:32:30.382937+0000","last_active":"2024-04-26T20:32:31.364193+0000","last_peered":"2024-04-26T20:32:31.364193+0000","last_clean":"2024-04-26T20:32:31.364193+0000","last_became_active":"2024-04-26T20:32:30.378828+0000","last_became_peered":"2024-04-26T20:32:30.378828+0000","last_unstale":"2024-04-26T20:32:31.364193+0000","last_undegraded":"2024-04-26T20:32:31.364193+0000","last_fullsized":"2024-04-26T20:32:31.364193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:33:41.685012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.656472+0000","last_change":"2024-04-26T20:32:25.280362+0000","last_active":"2024-04-26T20:32:51.656472+0000","last_peered":"2024-04-26T20:32:51.656472+0000","last_clean":"2024-04-26T20:32:51.656472+0000","last_became_active":"2024-04-26T20:32:25.279975+0000","last_became_peered":"2024-04-26T20:32:25.279975+0000","last_unstale":"2024-04-26T20:32:51.656472+0000","last_undegraded":"2024-04-26T20:32:51.656472+0000","last_fullsized":"2024-04-26T20:32:51.656472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:08.150635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548433+0000","last_change":"2024-04-26T20:32:26.231231+0000","last_active":"2024-04-26T20:32:31.548433+0000","last_peered":"2024-04-26T20:32:31.548433+0000","last_clean":"2024-04-26T20:32:31.548433+0000","last_became_active":"2024-04-26T20:32:26.230831+0000","last_became_peered":"2024-04-26T20:32:26.230831+0000","last_unstale":"2024-04-26T20:32:31.548433+0000","last_undegraded":"2024-04-26T20:32:31.548433+0000","last_fullsized":"2024-04-26T20:32:31.548433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:03:45.272542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549920+0000","last_change":"2024-04-26T20:32:30.382633+0000","last_active":"2024-04-26T20:32:31.549920+0000","last_peered":"2024-04-26T20:32:31.549920+0000","last_clean":"2024-04-26T20:32:31.549920+0000","last_became_active":"2024-04-26T20:32:30.379821+0000","last_became_peered":"2024-04-26T20:32:30.379821+0000","last_unstale":"2024-04-26T20:32:31.549920+0000","last_undegraded":"2024-04-26T20:32:31.549920+0000","last_fullsized":"2024-04-26T20:32:31.549920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:58:16.888172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.652483+0000","last_change":"2024-04-26T20:32:25.264844+0000","last_active":"2024-04-26T20:32:51.652483+0000","last_peered":"2024-04-26T20:32:51.652483+0000","last_clean":"2024-04-26T20:32:51.652483+0000","last_became_active":"2024-04-26T20:32:25.264097+0000","last_became_peered":"2024-04-26T20:32:25.264097+0000","last_unstale":"2024-04-26T20:32:51.652483+0000","last_undegraded":"2024-04-26T20:32:51.652483+0000","last_fullsized":"2024-04-26T20:32:51.652483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:09.308656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.368581+0000","last_change":"2024-04-26T20:32:26.248211+0000","last_active":"2024-04-26T20:32:31.368581+0000","last_peered":"2024-04-26T20:32:31.368581+0000","last_clean":"2024-04-26T20:32:31.368581+0000","last_became_active":"2024-04-26T20:32:26.247812+0000","last_became_peered":"2024-04-26T20:32:26.247812+0000","last_unstale":"2024-04-26T20:32:31.368581+0000","last_undegraded":"2024-04-26T20:32:31.368581+0000","last_fullsized":"2024-04-26T20:32:31.368581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:59:05.448957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550692+0000","last_change":"2024-04-26T20:32:30.377547+0000","last_active":"2024-04-26T20:32:31.550692+0000","last_peered":"2024-04-26T20:32:31.550692+0000","last_clean":"2024-04-26T20:32:31.550692+0000","last_became_active":"2024-04-26T20:32:30.377139+0000","last_became_peered":"2024-04-26T20:32:30.377139+0000","last_unstale":"2024-04-26T20:32:31.550692+0000","last_undegraded":"2024-04-26T20:32:31.550692+0000","last_fullsized":"2024-04-26T20:32:31.550692+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:41.970710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.655702+0000","last_change":"2024-04-26T20:32:25.284996+0000","last_active":"2024-04-26T20:32:51.655702+0000","last_peered":"2024-04-26T20:32:51.655702+0000","last_clean":"2024-04-26T20:32:51.655702+0000","last_became_active":"2024-04-26T20:32:25.284702+0000","last_became_peered":"2024-04-26T20:32:25.284702+0000","last_unstale":"2024-04-26T20:32:51.655702+0000","last_undegraded":"2024-04-26T20:32:51.655702+0000","last_fullsized":"2024-04-26T20:32:51.655702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:39.831522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548777+0000","last_change":"2024-04-26T20:32:26.230357+0000","last_active":"2024-04-26T20:32:31.548777+0000","last_peered":"2024-04-26T20:32:31.548777+0000","last_clean":"2024-04-26T20:32:31.548777+0000","last_became_active":"2024-04-26T20:32:26.229831+0000","last_became_peered":"2024-04-26T20:32:26.229831+0000","last_unstale":"2024-04-26T20:32:31.548777+0000","last_undegraded":"2024-04-26T20:32:31.548777+0000","last_fullsized":"2024-04-26T20:32:31.548777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:20:12.452768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550597+0000","last_change":"2024-04-26T20:32:30.383469+0000","last_active":"2024-04-26T20:32:31.550597+0000","last_peered":"2024-04-26T20:32:31.550597+0000","last_clean":"2024-04-26T20:32:31.550597+0000","last_became_active":"2024-04-26T20:32:30.382416+0000","last_became_peered":"2024-04-26T20:32:30.382416+0000","last_unstale":"2024-04-26T20:32:31.550597+0000","last_undegraded":"2024-04-26T20:32:31.550597+0000","last_fullsized":"2024-04-26T20:32:31.550597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:55.254214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.654974+0000","last_change":"2024-04-26T20:32:25.264433+0000","last_active":"2024-04-26T20:32:51.654974+0000","last_peered":"2024-04-26T20:32:51.654974+0000","last_clean":"2024-04-26T20:32:51.654974+0000","last_became_active":"2024-04-26T20:32:25.264104+0000","last_became_peered":"2024-04-26T20:32:25.264104+0000","last_unstale":"2024-04-26T20:32:51.654974+0000","last_undegraded":"2024-04-26T20:32:51.654974+0000","last_fullsized":"2024-04-26T20:32:51.654974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.030082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549425+0000","last_change":"2024-04-26T20:32:26.219704+0000","last_active":"2024-04-26T20:32:31.549425+0000","last_peered":"2024-04-26T20:32:31.549425+0000","last_clean":"2024-04-26T20:32:31.549425+0000","last_became_active":"2024-04-26T20:32:26.219397+0000","last_became_peered":"2024-04-26T20:32:26.219397+0000","last_unstale":"2024-04-26T20:32:31.549425+0000","last_undegraded":"2024-04-26T20:32:31.549425+0000","last_fullsized":"2024-04-26T20:32:31.549425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:04.570044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551493+0000","last_change":"2024-04-26T20:32:30.388828+0000","last_active":"2024-04-26T20:32:31.551493+0000","last_peered":"2024-04-26T20:32:31.551493+0000","last_clean":"2024-04-26T20:32:31.551493+0000","last_became_active":"2024-04-26T20:32:30.387748+0000","last_became_peered":"2024-04-26T20:32:30.387748+0000","last_unstale":"2024-04-26T20:32:31.551493+0000","last_undegraded":"2024-04-26T20:32:31.551493+0000","last_fullsized":"2024-04-26T20:32:31.551493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:53:09.900151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550046+0000","last_change":"2024-04-26T20:32:25.271910+0000","last_active":"2024-04-26T20:32:31.550046+0000","last_peered":"2024-04-26T20:32:31.550046+0000","last_clean":"2024-04-26T20:32:31.550046+0000","last_became_active":"2024-04-26T20:32:25.271572+0000","last_became_peered":"2024-04-26T20:32:25.271572+0000","last_unstale":"2024-04-26T20:32:31.550046+0000","last_undegraded":"2024-04-26T20:32:31.550046+0000","last_fullsized":"2024-04-26T20:32:31.550046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:32:59.013229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551909+0000","last_change":"2024-04-26T20:32:26.237691+0000","last_active":"2024-04-26T20:32:31.551909+0000","last_peered":"2024-04-26T20:32:31.551909+0000","last_clean":"2024-04-26T20:32:31.551909+0000","last_became_active":"2024-04-26T20:32:26.237338+0000","last_became_peered":"2024-04-26T20:32:26.237338+0000","last_unstale":"2024-04-26T20:32:31.551909+0000","last_undegraded":"2024-04-26T20:32:31.551909+0000","last_fullsized":"2024-04-26T20:32:31.551909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:51.782689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551138+0000","last_change":"2024-04-26T20:32:30.388485+0000","last_active":"2024-04-26T20:32:31.551138+0000","last_peered":"2024-04-26T20:32:31.551138+0000","last_clean":"2024-04-26T20:32:31.551138+0000","last_became_active":"2024-04-26T20:32:30.388074+0000","last_became_peered":"2024-04-26T20:32:30.388074+0000","last_unstale":"2024-04-26T20:32:31.551138+0000","last_undegraded":"2024-04-26T20:32:31.551138+0000","last_fullsized":"2024-04-26T20:32:31.551138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:52.792551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.658845+0000","last_change":"2024-04-26T20:32:25.271589+0000","last_active":"2024-04-26T20:32:51.658845+0000","last_peered":"2024-04-26T20:32:51.658845+0000","last_clean":"2024-04-26T20:32:51.658845+0000","last_became_active":"2024-04-26T20:32:25.271260+0000","last_became_peered":"2024-04-26T20:32:25.271260+0000","last_unstale":"2024-04-26T20:32:51.658845+0000","last_undegraded":"2024-04-26T20:32:51.658845+0000","last_fullsized":"2024-04-26T20:32:51.658845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:30.978561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361096+0000","last_change":"2024-04-26T20:32:26.240158+0000","last_active":"2024-04-26T20:32:31.361096+0000","last_peered":"2024-04-26T20:32:31.361096+0000","last_clean":"2024-04-26T20:32:31.361096+0000","last_became_active":"2024-04-26T20:32:26.239831+0000","last_became_peered":"2024-04-26T20:32:26.239831+0000","last_unstale":"2024-04-26T20:32:31.361096+0000","last_undegraded":"2024-04-26T20:32:31.361096+0000","last_fullsized":"2024-04-26T20:32:31.361096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:07.870097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551073+0000","last_change":"2024-04-26T20:32:30.392090+0000","last_active":"2024-04-26T20:32:31.551073+0000","last_peered":"2024-04-26T20:32:31.551073+0000","last_clean":"2024-04-26T20:32:31.551073+0000","last_became_active":"2024-04-26T20:32:30.390404+0000","last_became_peered":"2024-04-26T20:32:30.390404+0000","last_unstale":"2024-04-26T20:32:31.551073+0000","last_undegraded":"2024-04-26T20:32:31.551073+0000","last_fullsized":"2024-04-26T20:32:31.551073+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:50:10.743133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:53.674608+0000","last_change":"2024-04-26T20:32:25.264291+0000","last_active":"2024-04-26T20:32:53.674608+0000","last_peered":"2024-04-26T20:32:53.674608+0000","last_clean":"2024-04-26T20:32:53.674608+0000","last_became_active":"2024-04-26T20:32:25.263901+0000","last_became_peered":"2024-04-26T20:32:25.263901+0000","last_unstale":"2024-04-26T20:32:53.674608+0000","last_undegraded":"2024-04-26T20:32:53.674608+0000","last_fullsized":"2024-04-26T20:32:53.674608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:23:28.851941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.360941+0000","last_change":"2024-04-26T20:32:26.221472+0000","last_active":"2024-04-26T20:32:31.360941+0000","last_peered":"2024-04-26T20:32:31.360941+0000","last_clean":"2024-04-26T20:32:31.360941+0000","last_became_active":"2024-04-26T20:32:26.221117+0000","last_became_peered":"2024-04-26T20:32:26.221117+0000","last_unstale":"2024-04-26T20:32:31.360941+0000","last_undegraded":"2024-04-26T20:32:31.360941+0000","last_fullsized":"2024-04-26T20:32:31.360941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:27:01.666749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.553818+0000","last_change":"2024-04-26T20:32:30.392000+0000","last_active":"2024-04-26T20:32:31.553818+0000","last_peered":"2024-04-26T20:32:31.553818+0000","last_clean":"2024-04-26T20:32:31.553818+0000","last_became_active":"2024-04-26T20:32:30.386933+0000","last_became_peered":"2024-04-26T20:32:30.386933+0000","last_unstale":"2024-04-26T20:32:31.553818+0000","last_undegraded":"2024-04-26T20:32:31.553818+0000","last_fullsized":"2024-04-26T20:32:31.553818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:23.666439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:53.679043+0000","last_change":"2024-04-26T20:32:25.276482+0000","last_active":"2024-04-26T20:32:53.679043+0000","last_peered":"2024-04-26T20:32:53.679043+0000","last_clean":"2024-04-26T20:32:53.679043+0000","last_became_active":"2024-04-26T20:32:25.276159+0000","last_became_peered":"2024-04-26T20:32:25.276159+0000","last_unstale":"2024-04-26T20:32:53.679043+0000","last_undegraded":"2024-04-26T20:32:53.679043+0000","last_fullsized":"2024-04-26T20:32:53.679043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:34.029524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549686+0000","last_change":"2024-04-26T20:32:26.225344+0000","last_active":"2024-04-26T20:32:31.549686+0000","last_peered":"2024-04-26T20:32:31.549686+0000","last_clean":"2024-04-26T20:32:31.549686+0000","last_became_active":"2024-04-26T20:32:26.225039+0000","last_became_peered":"2024-04-26T20:32:26.225039+0000","last_unstale":"2024-04-26T20:32:31.549686+0000","last_undegraded":"2024-04-26T20:32:31.549686+0000","last_fullsized":"2024-04-26T20:32:31.549686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:08.680765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549624+0000","last_change":"2024-04-26T20:32:30.389274+0000","last_active":"2024-04-26T20:32:31.549624+0000","last_peered":"2024-04-26T20:32:31.549624+0000","last_clean":"2024-04-26T20:32:31.549624+0000","last_became_active":"2024-04-26T20:32:30.388527+0000","last_became_peered":"2024-04-26T20:32:30.388527+0000","last_unstale":"2024-04-26T20:32:31.549624+0000","last_undegraded":"2024-04-26T20:32:31.549624+0000","last_fullsized":"2024-04-26T20:32:31.549624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:14.689223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549588+0000","last_change":"2024-04-26T20:32:25.267338+0000","last_active":"2024-04-26T20:32:31.549588+0000","last_peered":"2024-04-26T20:32:31.549588+0000","last_clean":"2024-04-26T20:32:31.549588+0000","last_became_active":"2024-04-26T20:32:25.266234+0000","last_became_peered":"2024-04-26T20:32:25.266234+0000","last_unstale":"2024-04-26T20:32:31.549588+0000","last_undegraded":"2024-04-26T20:32:31.549588+0000","last_fullsized":"2024-04-26T20:32:31.549588+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:55.603147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551257+0000","last_change":"2024-04-26T20:32:26.222686+0000","last_active":"2024-04-26T20:32:31.551257+0000","last_peered":"2024-04-26T20:32:31.551257+0000","last_clean":"2024-04-26T20:32:31.551257+0000","last_became_active":"2024-04-26T20:32:26.222364+0000","last_became_peered":"2024-04-26T20:32:26.222364+0000","last_unstale":"2024-04-26T20:32:31.551257+0000","last_undegraded":"2024-04-26T20:32:31.551257+0000","last_fullsized":"2024-04-26T20:32:31.551257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:56.170913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548918+0000","last_change":"2024-04-26T20:32:30.414367+0000","last_active":"2024-04-26T20:32:31.548918+0000","last_peered":"2024-04-26T20:32:31.548918+0000","last_clean":"2024-04-26T20:32:31.548918+0000","last_became_active":"2024-04-26T20:32:30.413889+0000","last_became_peered":"2024-04-26T20:32:30.413889+0000","last_unstale":"2024-04-26T20:32:31.548918+0000","last_undegraded":"2024-04-26T20:32:31.548918+0000","last_fullsized":"2024-04-26T20:32:31.548918+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:11.336500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550321+0000","last_change":"2024-04-26T20:32:25.286408+0000","last_active":"2024-04-26T20:32:31.550321+0000","last_peered":"2024-04-26T20:32:31.550321+0000","last_clean":"2024-04-26T20:32:31.550321+0000","last_became_active":"2024-04-26T20:32:25.285986+0000","last_became_peered":"2024-04-26T20:32:25.285986+0000","last_unstale":"2024-04-26T20:32:31.550321+0000","last_undegraded":"2024-04-26T20:32:31.550321+0000","last_fullsized":"2024-04-26T20:32:31.550321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:47.757720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550746+0000","last_change":"2024-04-26T20:32:26.250459+0000","last_active":"2024-04-26T20:32:31.550746+0000","last_peered":"2024-04-26T20:32:31.550746+0000","last_clean":"2024-04-26T20:32:31.550746+0000","last_became_active":"2024-04-26T20:32:26.250154+0000","last_became_peered":"2024-04-26T20:32:26.250154+0000","last_unstale":"2024-04-26T20:32:31.550746+0000","last_undegraded":"2024-04-26T20:32:31.550746+0000","last_fullsized":"2024-04-26T20:32:31.550746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:47.185067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.552702+0000","last_change":"2024-04-26T20:32:30.395258+0000","last_active":"2024-04-26T20:32:31.552702+0000","last_peered":"2024-04-26T20:32:31.552702+0000","last_clean":"2024-04-26T20:32:31.552702+0000","last_became_active":"2024-04-26T20:32:30.392452+0000","last_became_peered":"2024-04-26T20:32:30.392452+0000","last_unstale":"2024-04-26T20:32:31.552702+0000","last_undegraded":"2024-04-26T20:32:31.552702+0000","last_fullsized":"2024-04-26T20:32:31.552702+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:19.883901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361095+0000","last_change":"2024-04-26T20:32:25.297361+0000","last_active":"2024-04-26T20:32:31.361095+0000","last_peered":"2024-04-26T20:32:31.361095+0000","last_clean":"2024-04-26T20:32:31.361095+0000","last_became_active":"2024-04-26T20:32:25.297125+0000","last_became_peered":"2024-04-26T20:32:25.297125+0000","last_unstale":"2024-04-26T20:32:31.361095+0000","last_undegraded":"2024-04-26T20:32:31.361095+0000","last_fullsized":"2024-04-26T20:32:31.361095+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:14:38.038683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550058+0000","last_change":"2024-04-26T20:32:26.249371+0000","last_active":"2024-04-26T20:32:31.550058+0000","last_peered":"2024-04-26T20:32:31.550058+0000","last_clean":"2024-04-26T20:32:31.550058+0000","last_became_active":"2024-04-26T20:32:26.249018+0000","last_became_peered":"2024-04-26T20:32:26.249018+0000","last_unstale":"2024-04-26T20:32:31.550058+0000","last_undegraded":"2024-04-26T20:32:31.550058+0000","last_fullsized":"2024-04-26T20:32:31.550058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:27.553928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.553674+0000","last_change":"2024-04-26T20:32:30.414854+0000","last_active":"2024-04-26T20:32:31.553674+0000","last_peered":"2024-04-26T20:32:31.553674+0000","last_clean":"2024-04-26T20:32:31.553674+0000","last_became_active":"2024-04-26T20:32:30.414374+0000","last_became_peered":"2024-04-26T20:32:30.414374+0000","last_unstale":"2024-04-26T20:32:31.553674+0000","last_undegraded":"2024-04-26T20:32:31.553674+0000","last_fullsized":"2024-04-26T20:32:31.553674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:32.071225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550965+0000","last_change":"2024-04-26T20:32:26.222681+0000","last_active":"2024-04-26T20:32:31.550965+0000","last_peered":"2024-04-26T20:32:31.550965+0000","last_clean":"2024-04-26T20:32:31.550965+0000","last_became_active":"2024-04-26T20:32:26.222099+0000","last_became_peered":"2024-04-26T20:32:26.222099+0000","last_unstale":"2024-04-26T20:32:31.550965+0000","last_undegraded":"2024-04-26T20:32:31.550965+0000","last_fullsized":"2024-04-26T20:32:31.550965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:35:21.958180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:52.669823+0000","last_change":"2024-04-26T20:32:25.276314+0000","last_active":"2024-04-26T20:32:52.669823+0000","last_peered":"2024-04-26T20:32:52.669823+0000","last_clean":"2024-04-26T20:32:52.669823+0000","last_became_active":"2024-04-26T20:32:25.275926+0000","last_became_peered":"2024-04-26T20:32:25.275926+0000","last_unstale":"2024-04-26T20:32:52.669823+0000","last_undegraded":"2024-04-26T20:32:52.669823+0000","last_fullsized":"2024-04-26T20:32:52.669823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:25.027777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361329+0000","last_change":"2024-04-26T20:32:30.385121+0000","last_active":"2024-04-26T20:32:31.361329+0000","last_peered":"2024-04-26T20:32:31.361329+0000","last_clean":"2024-04-26T20:32:31.361329+0000","last_became_active":"2024-04-26T20:32:30.384697+0000","last_became_peered":"2024-04-26T20:32:30.384697+0000","last_unstale":"2024-04-26T20:32:31.361329+0000","last_undegraded":"2024-04-26T20:32:31.361329+0000","last_fullsized":"2024-04-26T20:32:31.361329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:00.725130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.363801+0000","last_change":"2024-04-26T20:32:26.227550+0000","last_active":"2024-04-26T20:32:31.363801+0000","last_peered":"2024-04-26T20:32:31.363801+0000","last_clean":"2024-04-26T20:32:31.363801+0000","last_became_active":"2024-04-26T20:32:26.224262+0000","last_became_peered":"2024-04-26T20:32:26.224262+0000","last_unstale":"2024-04-26T20:32:31.363801+0000","last_undegraded":"2024-04-26T20:32:31.363801+0000","last_fullsized":"2024-04-26T20:32:31.363801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:01.413870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:53.681793+0000","last_change":"2024-04-26T20:32:25.274253+0000","last_active":"2024-04-26T20:32:53.681793+0000","last_peered":"2024-04-26T20:32:53.681793+0000","last_clean":"2024-04-26T20:32:53.681793+0000","last_became_active":"2024-04-26T20:32:25.273956+0000","last_became_peered":"2024-04-26T20:32:25.273956+0000","last_unstale":"2024-04-26T20:32:53.681793+0000","last_undegraded":"2024-04-26T20:32:53.681793+0000","last_fullsized":"2024-04-26T20:32:53.681793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:23:27.832135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":348,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.360757+0000","last_change":"2024-04-26T20:31:05.644347+0000","last_active":"2024-04-26T20:32:31.360757+0000","last_peered":"2024-04-26T20:32:31.360757+0000","last_clean":"2024-04-26T20:32:31.360757+0000","last_became_active":"2024-04-26T20:31:05.643888+0000","last_became_peered":"2024-04-26T20:31:05.643888+0000","last_unstale":"2024-04-26T20:32:31.360757+0000","last_undegraded":"2024-04-26T20:32:31.360757+0000","last_fullsized":"2024-04-26T20:32:31.360757+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:27:11.501328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:27:11.501328+0000","last_clean_scrub_stamp":"2024-04-26T20:27:11.501328+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:08.802687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550797+0000","last_change":"2024-04-26T20:32:30.414972+0000","last_active":"2024-04-26T20:32:31.550797+0000","last_peered":"2024-04-26T20:32:31.550797+0000","last_clean":"2024-04-26T20:32:31.550797+0000","last_became_active":"2024-04-26T20:32:30.414542+0000","last_became_peered":"2024-04-26T20:32:30.414542+0000","last_unstale":"2024-04-26T20:32:31.550797+0000","last_undegraded":"2024-04-26T20:32:31.550797+0000","last_fullsized":"2024-04-26T20:32:31.550797+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:49:06.242986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550900+0000","last_change":"2024-04-26T20:32:26.239936+0000","last_active":"2024-04-26T20:32:31.550900+0000","last_peered":"2024-04-26T20:32:31.550900+0000","last_clean":"2024-04-26T20:32:31.550900+0000","last_became_active":"2024-04-26T20:32:26.239583+0000","last_became_peered":"2024-04-26T20:32:26.239583+0000","last_unstale":"2024-04-26T20:32:31.550900+0000","last_undegraded":"2024-04-26T20:32:31.550900+0000","last_fullsized":"2024-04-26T20:32:31.550900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:57.183460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:52.664139+0000","last_change":"2024-04-26T20:32:25.267876+0000","last_active":"2024-04-26T20:32:52.664139+0000","last_peered":"2024-04-26T20:32:52.664139+0000","last_clean":"2024-04-26T20:32:52.664139+0000","last_became_active":"2024-04-26T20:32:25.267573+0000","last_became_peered":"2024-04-26T20:32:25.267573+0000","last_unstale":"2024-04-26T20:32:52.664139+0000","last_undegraded":"2024-04-26T20:32:52.664139+0000","last_fullsized":"2024-04-26T20:32:52.664139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:43:55.630426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550797+0000","last_change":"2024-04-26T20:32:30.383364+0000","last_active":"2024-04-26T20:32:31.550797+0000","last_peered":"2024-04-26T20:32:31.550797+0000","last_clean":"2024-04-26T20:32:31.550797+0000","last_became_active":"2024-04-26T20:32:30.382875+0000","last_became_peered":"2024-04-26T20:32:30.382875+0000","last_unstale":"2024-04-26T20:32:31.550797+0000","last_undegraded":"2024-04-26T20:32:31.550797+0000","last_fullsized":"2024-04-26T20:32:31.550797+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:55.112888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:53.681035+0000","last_change":"2024-04-26T20:32:25.280508+0000","last_active":"2024-04-26T20:32:53.681035+0000","last_peered":"2024-04-26T20:32:53.681035+0000","last_clean":"2024-04-26T20:32:53.681035+0000","last_became_active":"2024-04-26T20:32:25.280265+0000","last_became_peered":"2024-04-26T20:32:25.280265+0000","last_unstale":"2024-04-26T20:32:53.681035+0000","last_undegraded":"2024-04-26T20:32:53.681035+0000","last_fullsized":"2024-04-26T20:32:53.681035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:16:50.427659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.552664+0000","last_change":"2024-04-26T20:32:26.255230+0000","last_active":"2024-04-26T20:32:31.552664+0000","last_peered":"2024-04-26T20:32:31.552664+0000","last_clean":"2024-04-26T20:32:31.552664+0000","last_became_active":"2024-04-26T20:32:26.254099+0000","last_became_peered":"2024-04-26T20:32:26.254099+0000","last_unstale":"2024-04-26T20:32:31.552664+0000","last_undegraded":"2024-04-26T20:32:31.552664+0000","last_fullsized":"2024-04-26T20:32:31.552664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:41.895816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.363425+0000","last_change":"2024-04-26T20:32:30.414688+0000","last_active":"2024-04-26T20:32:31.363425+0000","last_peered":"2024-04-26T20:32:31.363425+0000","last_clean":"2024-04-26T20:32:31.363425+0000","last_became_active":"2024-04-26T20:32:30.414052+0000","last_became_peered":"2024-04-26T20:32:30.414052+0000","last_unstale":"2024-04-26T20:32:31.363425+0000","last_undegraded":"2024-04-26T20:32:31.363425+0000","last_fullsized":"2024-04-26T20:32:31.363425+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:24.943559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:53.683430+0000","last_change":"2024-04-26T20:32:25.298609+0000","last_active":"2024-04-26T20:32:53.683430+0000","last_peered":"2024-04-26T20:32:53.683430+0000","last_clean":"2024-04-26T20:32:53.683430+0000","last_became_active":"2024-04-26T20:32:25.298155+0000","last_became_peered":"2024-04-26T20:32:25.298155+0000","last_unstale":"2024-04-26T20:32:53.683430+0000","last_undegraded":"2024-04-26T20:32:53.683430+0000","last_fullsized":"2024-04-26T20:32:53.683430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:37.130994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.365759+0000","last_change":"2024-04-26T20:32:26.231203+0000","last_active":"2024-04-26T20:32:31.365759+0000","last_peered":"2024-04-26T20:32:31.365759+0000","last_clean":"2024-04-26T20:32:31.365759+0000","last_became_active":"2024-04-26T20:32:26.230775+0000","last_became_peered":"2024-04-26T20:32:26.230775+0000","last_unstale":"2024-04-26T20:32:31.365759+0000","last_undegraded":"2024-04-26T20:32:31.365759+0000","last_fullsized":"2024-04-26T20:32:31.365759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:40.557744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549252+0000","last_change":"2024-04-26T20:32:30.375703+0000","last_active":"2024-04-26T20:32:31.549252+0000","last_peered":"2024-04-26T20:32:31.549252+0000","last_clean":"2024-04-26T20:32:31.549252+0000","last_became_active":"2024-04-26T20:32:30.375292+0000","last_became_peered":"2024-04-26T20:32:30.375292+0000","last_unstale":"2024-04-26T20:32:31.549252+0000","last_undegraded":"2024-04-26T20:32:31.549252+0000","last_fullsized":"2024-04-26T20:32:31.549252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:39.747426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.658100+0000","last_change":"2024-04-26T20:32:25.290908+0000","last_active":"2024-04-26T20:32:51.658100+0000","last_peered":"2024-04-26T20:32:51.658100+0000","last_clean":"2024-04-26T20:32:51.658100+0000","last_became_active":"2024-04-26T20:32:25.290655+0000","last_became_peered":"2024-04-26T20:32:25.290655+0000","last_unstale":"2024-04-26T20:32:51.658100+0000","last_undegraded":"2024-04-26T20:32:51.658100+0000","last_fullsized":"2024-04-26T20:32:51.658100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:28:42.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.368104+0000","last_change":"2024-04-26T20:32:26.246554+0000","last_active":"2024-04-26T20:32:31.368104+0000","last_peered":"2024-04-26T20:32:31.368104+0000","last_clean":"2024-04-26T20:32:31.368104+0000","last_became_active":"2024-04-26T20:32:26.246279+0000","last_became_peered":"2024-04-26T20:32:26.246279+0000","last_unstale":"2024-04-26T20:32:31.368104+0000","last_undegraded":"2024-04-26T20:32:31.368104+0000","last_fullsized":"2024-04-26T20:32:31.368104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:02.561410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550615+0000","last_change":"2024-04-26T20:32:30.414622+0000","last_active":"2024-04-26T20:32:31.550615+0000","last_peered":"2024-04-26T20:32:31.550615+0000","last_clean":"2024-04-26T20:32:31.550615+0000","last_became_active":"2024-04-26T20:32:30.414176+0000","last_became_peered":"2024-04-26T20:32:30.414176+0000","last_unstale":"2024-04-26T20:32:31.550615+0000","last_undegraded":"2024-04-26T20:32:31.550615+0000","last_fullsized":"2024-04-26T20:32:31.550615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:15.011512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.653091+0000","last_change":"2024-04-26T20:32:25.275220+0000","last_active":"2024-04-26T20:32:51.653091+0000","last_peered":"2024-04-26T20:32:51.653091+0000","last_clean":"2024-04-26T20:32:51.653091+0000","last_became_active":"2024-04-26T20:32:25.274990+0000","last_became_peered":"2024-04-26T20:32:25.274990+0000","last_unstale":"2024-04-26T20:32:51.653091+0000","last_undegraded":"2024-04-26T20:32:51.653091+0000","last_fullsized":"2024-04-26T20:32:51.653091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:58.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548678+0000","last_change":"2024-04-26T20:32:26.218481+0000","last_active":"2024-04-26T20:32:31.548678+0000","last_peered":"2024-04-26T20:32:31.548678+0000","last_clean":"2024-04-26T20:32:31.548678+0000","last_became_active":"2024-04-26T20:32:26.218146+0000","last_became_peered":"2024-04-26T20:32:26.218146+0000","last_unstale":"2024-04-26T20:32:31.548678+0000","last_undegraded":"2024-04-26T20:32:31.548678+0000","last_fullsized":"2024-04-26T20:32:31.548678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:37.310945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551737+0000","last_change":"2024-04-26T20:32:30.383815+0000","last_active":"2024-04-26T20:32:31.551737+0000","last_peered":"2024-04-26T20:32:31.551737+0000","last_clean":"2024-04-26T20:32:31.551737+0000","last_became_active":"2024-04-26T20:32:30.382818+0000","last_became_peered":"2024-04-26T20:32:30.382818+0000","last_unstale":"2024-04-26T20:32:31.551737+0000","last_undegraded":"2024-04-26T20:32:31.551737+0000","last_fullsized":"2024-04-26T20:32:31.551737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:47.839781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.658138+0000","last_change":"2024-04-26T20:32:25.291527+0000","last_active":"2024-04-26T20:32:51.658138+0000","last_peered":"2024-04-26T20:32:51.658138+0000","last_clean":"2024-04-26T20:32:51.658138+0000","last_became_active":"2024-04-26T20:32:25.285353+0000","last_became_peered":"2024-04-26T20:32:25.285353+0000","last_unstale":"2024-04-26T20:32:51.658138+0000","last_undegraded":"2024-04-26T20:32:51.658138+0000","last_fullsized":"2024-04-26T20:32:51.658138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:28.927822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550304+0000","last_change":"2024-04-26T20:32:26.241064+0000","last_active":"2024-04-26T20:32:31.550304+0000","last_peered":"2024-04-26T20:32:31.550304+0000","last_clean":"2024-04-26T20:32:31.550304+0000","last_became_active":"2024-04-26T20:32:26.240673+0000","last_became_peered":"2024-04-26T20:32:26.240673+0000","last_unstale":"2024-04-26T20:32:31.550304+0000","last_undegraded":"2024-04-26T20:32:31.550304+0000","last_fullsized":"2024-04-26T20:32:31.550304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:19:47.777751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550107+0000","last_change":"2024-04-26T20:32:30.388465+0000","last_active":"2024-04-26T20:32:31.550107+0000","last_peered":"2024-04-26T20:32:31.550107+0000","last_clean":"2024-04-26T20:32:31.550107+0000","last_became_active":"2024-04-26T20:32:30.388038+0000","last_became_peered":"2024-04-26T20:32:30.388038+0000","last_unstale":"2024-04-26T20:32:31.550107+0000","last_undegraded":"2024-04-26T20:32:31.550107+0000","last_fullsized":"2024-04-26T20:32:31.550107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:38:39.596970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549755+0000","last_change":"2024-04-26T20:32:25.271914+0000","last_active":"2024-04-26T20:32:31.549755+0000","last_peered":"2024-04-26T20:32:31.549755+0000","last_clean":"2024-04-26T20:32:31.549755+0000","last_became_active":"2024-04-26T20:32:25.271567+0000","last_became_peered":"2024-04-26T20:32:25.271567+0000","last_unstale":"2024-04-26T20:32:31.549755+0000","last_undegraded":"2024-04-26T20:32:31.549755+0000","last_fullsized":"2024-04-26T20:32:31.549755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:02.182470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.364761+0000","last_change":"2024-04-26T20:32:26.245344+0000","last_active":"2024-04-26T20:32:31.364761+0000","last_peered":"2024-04-26T20:32:31.364761+0000","last_clean":"2024-04-26T20:32:31.364761+0000","last_became_active":"2024-04-26T20:32:26.245032+0000","last_became_peered":"2024-04-26T20:32:26.245032+0000","last_unstale":"2024-04-26T20:32:31.364761+0000","last_undegraded":"2024-04-26T20:32:31.364761+0000","last_fullsized":"2024-04-26T20:32:31.364761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:12:51.368935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549938+0000","last_change":"2024-04-26T20:32:30.414408+0000","last_active":"2024-04-26T20:32:31.549938+0000","last_peered":"2024-04-26T20:32:31.549938+0000","last_clean":"2024-04-26T20:32:31.549938+0000","last_became_active":"2024-04-26T20:32:30.413892+0000","last_became_peered":"2024-04-26T20:32:30.413892+0000","last_unstale":"2024-04-26T20:32:31.549938+0000","last_undegraded":"2024-04-26T20:32:31.549938+0000","last_fullsized":"2024-04-26T20:32:31.549938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:53.332840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.652652+0000","last_change":"2024-04-26T20:32:25.265440+0000","last_active":"2024-04-26T20:32:51.652652+0000","last_peered":"2024-04-26T20:32:51.652652+0000","last_clean":"2024-04-26T20:32:51.652652+0000","last_became_active":"2024-04-26T20:32:25.265129+0000","last_became_peered":"2024-04-26T20:32:25.265129+0000","last_unstale":"2024-04-26T20:32:51.652652+0000","last_undegraded":"2024-04-26T20:32:51.652652+0000","last_fullsized":"2024-04-26T20:32:51.652652+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:49.475009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361178+0000","last_change":"2024-04-26T20:32:26.234477+0000","last_active":"2024-04-26T20:32:31.361178+0000","last_peered":"2024-04-26T20:32:31.361178+0000","last_clean":"2024-04-26T20:32:31.361178+0000","last_became_active":"2024-04-26T20:32:26.234103+0000","last_became_peered":"2024-04-26T20:32:26.234103+0000","last_unstale":"2024-04-26T20:32:31.361178+0000","last_undegraded":"2024-04-26T20:32:31.361178+0000","last_fullsized":"2024-04-26T20:32:31.361178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:54.098860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548855+0000","last_change":"2024-04-26T20:32:30.391466+0000","last_active":"2024-04-26T20:32:31.548855+0000","last_peered":"2024-04-26T20:32:31.548855+0000","last_clean":"2024-04-26T20:32:31.548855+0000","last_became_active":"2024-04-26T20:32:30.391037+0000","last_became_peered":"2024-04-26T20:32:30.391037+0000","last_unstale":"2024-04-26T20:32:31.548855+0000","last_undegraded":"2024-04-26T20:32:31.548855+0000","last_fullsized":"2024-04-26T20:32:31.548855+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:39:16.305520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550548+0000","last_change":"2024-04-26T20:32:25.281688+0000","last_active":"2024-04-26T20:32:31.550548+0000","last_peered":"2024-04-26T20:32:31.550548+0000","last_clean":"2024-04-26T20:32:31.550548+0000","last_became_active":"2024-04-26T20:32:25.281430+0000","last_became_peered":"2024-04-26T20:32:25.281430+0000","last_unstale":"2024-04-26T20:32:31.550548+0000","last_undegraded":"2024-04-26T20:32:31.550548+0000","last_fullsized":"2024-04-26T20:32:31.550548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:52.785532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361528+0000","last_change":"2024-04-26T20:32:26.226468+0000","last_active":"2024-04-26T20:32:31.361528+0000","last_peered":"2024-04-26T20:32:31.361528+0000","last_clean":"2024-04-26T20:32:31.361528+0000","last_became_active":"2024-04-26T20:32:26.225399+0000","last_became_peered":"2024-04-26T20:32:26.225399+0000","last_unstale":"2024-04-26T20:32:31.361528+0000","last_undegraded":"2024-04-26T20:32:31.361528+0000","last_fullsized":"2024-04-26T20:32:31.361528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:17:11.787658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366950+0000","last_change":"2024-04-26T20:32:30.383646+0000","last_active":"2024-04-26T20:32:31.366950+0000","last_peered":"2024-04-26T20:32:31.366950+0000","last_clean":"2024-04-26T20:32:31.366950+0000","last_became_active":"2024-04-26T20:32:30.381648+0000","last_became_peered":"2024-04-26T20:32:30.381648+0000","last_unstale":"2024-04-26T20:32:31.366950+0000","last_undegraded":"2024-04-26T20:32:31.366950+0000","last_fullsized":"2024-04-26T20:32:31.366950+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:28.036984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551284+0000","last_change":"2024-04-26T20:32:25.275597+0000","last_active":"2024-04-26T20:32:31.551284+0000","last_peered":"2024-04-26T20:32:31.551284+0000","last_clean":"2024-04-26T20:32:31.551284+0000","last_became_active":"2024-04-26T20:32:25.275256+0000","last_became_peered":"2024-04-26T20:32:25.275256+0000","last_unstale":"2024-04-26T20:32:31.551284+0000","last_undegraded":"2024-04-26T20:32:31.551284+0000","last_fullsized":"2024-04-26T20:32:31.551284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:38:20.354879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361187+0000","last_change":"2024-04-26T20:32:26.236885+0000","last_active":"2024-04-26T20:32:31.361187+0000","last_peered":"2024-04-26T20:32:31.361187+0000","last_clean":"2024-04-26T20:32:31.361187+0000","last_became_active":"2024-04-26T20:32:26.236581+0000","last_became_peered":"2024-04-26T20:32:26.236581+0000","last_unstale":"2024-04-26T20:32:31.361187+0000","last_undegraded":"2024-04-26T20:32:31.361187+0000","last_fullsized":"2024-04-26T20:32:31.361187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:17:35.082564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549096+0000","last_change":"2024-04-26T20:32:30.380156+0000","last_active":"2024-04-26T20:32:31.549096+0000","last_peered":"2024-04-26T20:32:31.549096+0000","last_clean":"2024-04-26T20:32:31.549096+0000","last_became_active":"2024-04-26T20:32:30.379240+0000","last_became_peered":"2024-04-26T20:32:30.379240+0000","last_unstale":"2024-04-26T20:32:31.549096+0000","last_undegraded":"2024-04-26T20:32:31.549096+0000","last_fullsized":"2024-04-26T20:32:31.549096+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:42:39.650923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.553339+0000","last_change":"2024-04-26T20:32:25.298850+0000","last_active":"2024-04-26T20:32:31.553339+0000","last_peered":"2024-04-26T20:32:31.553339+0000","last_clean":"2024-04-26T20:32:31.553339+0000","last_became_active":"2024-04-26T20:32:25.298171+0000","last_became_peered":"2024-04-26T20:32:25.298171+0000","last_unstale":"2024-04-26T20:32:31.553339+0000","last_undegraded":"2024-04-26T20:32:31.553339+0000","last_fullsized":"2024-04-26T20:32:31.553339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:17:03.723432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361349+0000","last_change":"2024-04-26T20:32:26.222873+0000","last_active":"2024-04-26T20:32:31.361349+0000","last_peered":"2024-04-26T20:32:31.361349+0000","last_clean":"2024-04-26T20:32:31.361349+0000","last_became_active":"2024-04-26T20:32:26.222487+0000","last_became_peered":"2024-04-26T20:32:26.222487+0000","last_unstale":"2024-04-26T20:32:31.361349+0000","last_undegraded":"2024-04-26T20:32:31.361349+0000","last_fullsized":"2024-04-26T20:32:31.361349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:17.521537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550408+0000","last_change":"2024-04-26T20:32:26.227092+0000","last_active":"2024-04-26T20:32:31.550408+0000","last_peered":"2024-04-26T20:32:31.550408+0000","last_clean":"2024-04-26T20:32:31.550408+0000","last_became_active":"2024-04-26T20:32:26.226748+0000","last_became_peered":"2024-04-26T20:32:26.226748+0000","last_unstale":"2024-04-26T20:32:31.550408+0000","last_undegraded":"2024-04-26T20:32:31.550408+0000","last_fullsized":"2024-04-26T20:32:31.550408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:25.141131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549875+0000","last_change":"2024-04-26T20:32:26.220388+0000","last_active":"2024-04-26T20:32:31.549875+0000","last_peered":"2024-04-26T20:32:31.549875+0000","last_clean":"2024-04-26T20:32:31.549875+0000","last_became_active":"2024-04-26T20:32:26.219999+0000","last_became_peered":"2024-04-26T20:32:26.219999+0000","last_unstale":"2024-04-26T20:32:31.549875+0000","last_undegraded":"2024-04-26T20:32:31.549875+0000","last_fullsized":"2024-04-26T20:32:31.549875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:11.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549631+0000","last_change":"2024-04-26T20:32:26.240122+0000","last_active":"2024-04-26T20:32:31.549631+0000","last_peered":"2024-04-26T20:32:31.549631+0000","last_clean":"2024-04-26T20:32:31.549631+0000","last_became_active":"2024-04-26T20:32:26.239746+0000","last_became_peered":"2024-04-26T20:32:26.239746+0000","last_unstale":"2024-04-26T20:32:31.549631+0000","last_undegraded":"2024-04-26T20:32:31.549631+0000","last_fullsized":"2024-04-26T20:32:31.549631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:10.243469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366722+0000","last_change":"2024-04-26T20:32:26.254171+0000","last_active":"2024-04-26T20:32:31.366722+0000","last_peered":"2024-04-26T20:32:31.366722+0000","last_clean":"2024-04-26T20:32:31.366722+0000","last_became_active":"2024-04-26T20:32:26.253775+0000","last_became_peered":"2024-04-26T20:32:26.253775+0000","last_unstale":"2024-04-26T20:32:31.366722+0000","last_undegraded":"2024-04-26T20:32:31.366722+0000","last_fullsized":"2024-04-26T20:32:31.366722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:19:57.968344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.552994+0000","last_change":"2024-04-26T20:32:26.256475+0000","last_active":"2024-04-26T20:32:31.552994+0000","last_peered":"2024-04-26T20:32:31.552994+0000","last_clean":"2024-04-26T20:32:31.552994+0000","last_became_active":"2024-04-26T20:32:26.256119+0000","last_became_peered":"2024-04-26T20:32:26.256119+0000","last_unstale":"2024-04-26T20:32:31.552994+0000","last_undegraded":"2024-04-26T20:32:31.552994+0000","last_fullsized":"2024-04-26T20:32:31.552994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:39:31.456077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550901+0000","last_change":"2024-04-26T20:32:26.229251+0000","last_active":"2024-04-26T20:32:31.550901+0000","last_peered":"2024-04-26T20:32:31.550901+0000","last_clean":"2024-04-26T20:32:31.550901+0000","last_became_active":"2024-04-26T20:32:26.228847+0000","last_became_peered":"2024-04-26T20:32:26.228847+0000","last_unstale":"2024-04-26T20:32:31.550901+0000","last_undegraded":"2024-04-26T20:32:31.550901+0000","last_fullsized":"2024-04-26T20:32:31.550901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:07:28.489681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550455+0000","last_change":"2024-04-26T20:32:26.223939+0000","last_active":"2024-04-26T20:32:31.550455+0000","last_peered":"2024-04-26T20:32:31.550455+0000","last_clean":"2024-04-26T20:32:31.550455+0000","last_became_active":"2024-04-26T20:32:26.223602+0000","last_became_peered":"2024-04-26T20:32:26.223602+0000","last_unstale":"2024-04-26T20:32:31.550455+0000","last_undegraded":"2024-04-26T20:32:31.550455+0000","last_fullsized":"2024-04-26T20:32:31.550455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:54.588411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366001+0000","last_change":"2024-04-26T20:32:26.229912+0000","last_active":"2024-04-26T20:32:31.366001+0000","last_peered":"2024-04-26T20:32:31.366001+0000","last_clean":"2024-04-26T20:32:31.366001+0000","last_became_active":"2024-04-26T20:32:26.228498+0000","last_became_peered":"2024-04-26T20:32:26.228498+0000","last_unstale":"2024-04-26T20:32:31.366001+0000","last_undegraded":"2024-04-26T20:32:31.366001+0000","last_fullsized":"2024-04-26T20:32:31.366001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:17:53.302225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550578+0000","last_change":"2024-04-26T20:32:26.232087+0000","last_active":"2024-04-26T20:32:31.550578+0000","last_peered":"2024-04-26T20:32:31.550578+0000","last_clean":"2024-04-26T20:32:31.550578+0000","last_became_active":"2024-04-26T20:32:26.231525+0000","last_became_peered":"2024-04-26T20:32:26.231525+0000","last_unstale":"2024-04-26T20:32:31.550578+0000","last_undegraded":"2024-04-26T20:32:31.550578+0000","last_fullsized":"2024-04-26T20:32:31.550578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:05.326866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.364004+0000","last_change":"2024-04-26T20:32:26.224341+0000","last_active":"2024-04-26T20:32:31.364004+0000","last_peered":"2024-04-26T20:32:31.364004+0000","last_clean":"2024-04-26T20:32:31.364004+0000","last_became_active":"2024-04-26T20:32:26.223888+0000","last_became_peered":"2024-04-26T20:32:26.223888+0000","last_unstale":"2024-04-26T20:32:31.364004+0000","last_undegraded":"2024-04-26T20:32:31.364004+0000","last_fullsized":"2024-04-26T20:32:31.364004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:21.103180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548415+0000","last_change":"2024-04-26T20:32:26.231255+0000","last_active":"2024-04-26T20:32:31.548415+0000","last_peered":"2024-04-26T20:32:31.548415+0000","last_clean":"2024-04-26T20:32:31.548415+0000","last_became_active":"2024-04-26T20:32:26.230823+0000","last_became_peered":"2024-04-26T20:32:26.230823+0000","last_unstale":"2024-04-26T20:32:31.548415+0000","last_undegraded":"2024-04-26T20:32:31.548415+0000","last_fullsized":"2024-04-26T20:32:31.548415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:34.066151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551796+0000","last_change":"2024-04-26T20:32:26.246452+0000","last_active":"2024-04-26T20:32:31.551796+0000","last_peered":"2024-04-26T20:32:31.551796+0000","last_clean":"2024-04-26T20:32:31.551796+0000","last_became_active":"2024-04-26T20:32:26.246109+0000","last_became_peered":"2024-04-26T20:32:26.246109+0000","last_unstale":"2024-04-26T20:32:31.551796+0000","last_undegraded":"2024-04-26T20:32:31.551796+0000","last_fullsized":"2024-04-26T20:32:31.551796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:42:21.617776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550782+0000","last_change":"2024-04-26T20:32:26.226274+0000","last_active":"2024-04-26T20:32:31.550782+0000","last_peered":"2024-04-26T20:32:31.550782+0000","last_clean":"2024-04-26T20:32:31.550782+0000","last_became_active":"2024-04-26T20:32:26.225751+0000","last_became_peered":"2024-04-26T20:32:26.225751+0000","last_unstale":"2024-04-26T20:32:31.550782+0000","last_undegraded":"2024-04-26T20:32:31.550782+0000","last_fullsized":"2024-04-26T20:32:31.550782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:39.050161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8326,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":22890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1072381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":482433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070501,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":477953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332137,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1068787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":481091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":477881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1605632,"data_stored":1069059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T20:32:57.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T20:32:57.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.669+0000 7fbca8c00640 1 -- 172.21.15.102:0/2947312069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbca4106720 msgr2=0x7fbca410cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.669+0000 7fbca8c00640 1 --2- 172.21.15.102:0/2947312069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbca4106720 0x7fbca410cfb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009ca0 tx=0x7fbc9c02a8c0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.669+0000 7fbca8c00640 1 -- 172.21.15.102:0/2947312069 shutdown_connections 2024-04-26T20:32:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.669+0000 7fbca8c00640 1 --2- 172.21.15.102:0/2947312069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbca4106720 0x7fbca410cfb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.669+0000 7fbca8c00640 1 --2- 172.21.15.102:0/2947312069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbca4105d60 0x7fbca41061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.669+0000 7fbca8c00640 1 --2- 172.21.15.102:0/2947312069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbca4104b70 0x7fbca4104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.669+0000 7fbca8c00640 1 -- 172.21.15.102:0/2947312069 >> 172.21.15.102:0/2947312069 conn(0x7fbca4100120 msgr2=0x7fbca4102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.669+0000 7fbca8c00640 1 -- 172.21.15.102:0/2947312069 shutdown_connections 2024-04-26T20:32:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.669+0000 7fbca8c00640 1 -- 172.21.15.102:0/2947312069 wait complete. 2024-04-26T20:32:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.670+0000 7fbca8c00640 1 Processor -- start 2024-04-26T20:32:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.670+0000 7fbca8c00640 1 -- start start 2024-04-26T20:32:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.670+0000 7fbca8c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbca4104b70 0x7fbca419ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.670+0000 7fbca8c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbca4105d60 0x7fbca419b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.670+0000 7fbca8c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbca4106720 0x7fbca41a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.670+0000 7fbca8c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbca41106d0 con 0x7fbca4106720 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.670+0000 7fbca8c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbca4110550 con 0x7fbca4105d60 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.670+0000 7fbca8c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbca4110850 con 0x7fbca4104b70 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.671+0000 7fbca2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbca4106720 0x7fbca41a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.671+0000 7fbca2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbca4106720 0x7fbca41a22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33160/0 (socket says 172.21.15.102:33160) 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.671+0000 7fbca2800640 1 -- 172.21.15.102:0/417463229 learned_addr learned my addr 172.21.15.102:0/417463229 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.671+0000 7fbca1400640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbca4105d60 0x7fbca419b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.671+0000 7fbca1e00640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbca4104b70 0x7fbca419ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.671+0000 7fbca2800640 1 -- 172.21.15.102:0/417463229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbca4104b70 msgr2=0x7fbca419ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.671+0000 7fbca2800640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbca4104b70 0x7fbca419ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.671+0000 7fbca2800640 1 -- 172.21.15.102:0/417463229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbca4105d60 msgr2=0x7fbca419b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.671+0000 7fbca2800640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbca4105d60 0x7fbca419b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.671+0000 7fbca2800640 1 -- 172.21.15.102:0/417463229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca41a2940 con 0x7fbca4106720 2024-04-26T20:32:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.672+0000 7fbca2800640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbca4106720 0x7fbca41a22a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c033040 tx=0x7fbc9c0316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.672+0000 7fbc9aa00640 1 -- 172.21.15.102:0/417463229 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c003b70 con 0x7fbca4106720 2024-04-26T20:32:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.672+0000 7fbc9aa00640 1 -- 172.21.15.102:0/417463229 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc9c003d10 con 0x7fbca4106720 2024-04-26T20:32:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.672+0000 7fbca8c00640 1 -- 172.21.15.102:0/417463229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca41a2bd0 con 0x7fbca4106720 2024-04-26T20:32:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.672+0000 7fbc9aa00640 1 -- 172.21.15.102:0/417463229 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c02fb40 con 0x7fbca4106720 2024-04-26T20:32:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.672+0000 7fbca8c00640 1 -- 172.21.15.102:0/417463229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca41a3120 con 0x7fbca4106720 2024-04-26T20:32:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.673+0000 7fbca8c00640 1 -- 172.21.15.102:0/417463229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca410be90 con 0x7fbca4106720 2024-04-26T20:32:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.675+0000 7fbc9aa00640 1 -- 172.21.15.102:0/417463229 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fbc9c004610 con 0x7fbca4106720 2024-04-26T20:32:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.677+0000 7fbc9aa00640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbc880a3c40 0x7fbc880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:32:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.677+0000 7fbc9aa00640 1 -- 172.21.15.102:0/417463229 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbc9c0f1b10 con 0x7fbca4106720 2024-04-26T20:32:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.677+0000 7fbca1e00640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbc880a3c40 0x7fbc880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:32:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.678+0000 7fbca1e00640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbc880a3c40 0x7fbc880a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbca419c200 tx=0x7fbc94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:32:57.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.681+0000 7fbc9aa00640 1 -- 172.21.15.102:0/417463229 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbc9c0be260 con 0x7fbca4106720 2024-04-26T20:32:57.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.883+0000 7fbca8c00640 1 -- 172.21.15.102:0/417463229 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbca4062d70 con 0x7fbc880a3c40 2024-04-26T20:32:57.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.910+0000 7fbc9aa00640 1 -- 172.21.15.102:0/417463229 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340333 (secure 0 0 0) 0x7fbca4062d70 con 0x7fbc880a3c40 2024-04-26T20:32:57.915 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:32:57.917 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-26T20:32:57.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 -- 172.21.15.102:0/417463229 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbc880a3c40 msgr2=0x7fbc880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:57.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbc880a3c40 0x7fbc880a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbca419c200 tx=0x7fbc94005f90 comp rx=0 tx=0).stop 2024-04-26T20:32:57.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 -- 172.21.15.102:0/417463229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbca4106720 msgr2=0x7fbca41a22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:32:57.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbca4106720 0x7fbca41a22a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c033040 tx=0x7fbc9c0316f0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 -- 172.21.15.102:0/417463229 shutdown_connections 2024-04-26T20:32:57.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbc880a3c40 0x7fbc880a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbca4106720 0x7fbca41a22a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbca4105d60 0x7fbca419b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 --2- 172.21.15.102:0/417463229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbca4104b70 0x7fbca419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:32:57.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 -- 172.21.15.102:0/417463229 >> 172.21.15.102:0/417463229 conn(0x7fbca4100120 msgr2=0x7fbca4109060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:32:57.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 -- 172.21.15.102:0/417463229 shutdown_connections 2024-04-26T20:32:57.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:32:57.916+0000 7fbca8c00640 1 -- 172.21.15.102:0/417463229 wait complete. 2024-04-26T20:32:57.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:57 smithi102 ceph-mon[28094]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:57.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:57 smithi102 ceph-mon[28094]: pgmap v333: 129 pgs: 129 active+clean; 585 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 9 op/s 2024-04-26T20:32:57.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:57 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2945108522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:57.937 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-04-26T20:32:56.525413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598694,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":340,"num_write_kb":2993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":350144,"kb_used_data":13472,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124591680,"statfs":{"total":1151940427776,"available":1151581880320,"internally_reserved":0,"allocated":13795328,"data_stored":7521062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":344634682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5467,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005876"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551581+0000","last_change":"2024-04-26T20:32:26.249504+0000","last_active":"2024-04-26T20:32:31.551581+0000","last_peered":"2024-04-26T20:32:31.551581+0000","last_clean":"2024-04-26T20:32:31.551581+0000","last_became_active":"2024-04-26T20:32:26.249188+0000","last_became_peered":"2024-04-26T20:32:26.249188+0000","last_unstale":"2024-04-26T20:32:31.551581+0000","last_undegraded":"2024-04-26T20:32:31.551581+0000","last_fullsized":"2024-04-26T20:32:31.551581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:28:15.906199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.367854+0000","last_change":"2024-04-26T20:32:26.246991+0000","last_active":"2024-04-26T20:32:31.367854+0000","last_peered":"2024-04-26T20:32:31.367854+0000","last_clean":"2024-04-26T20:32:31.367854+0000","last_became_active":"2024-04-26T20:32:26.245552+0000","last_became_peered":"2024-04-26T20:32:26.245552+0000","last_unstale":"2024-04-26T20:32:31.367854+0000","last_undegraded":"2024-04-26T20:32:31.367854+0000","last_fullsized":"2024-04-26T20:32:31.367854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:59:46.487659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366359+0000","last_change":"2024-04-26T20:32:26.241451+0000","last_active":"2024-04-26T20:32:31.366359+0000","last_peered":"2024-04-26T20:32:31.366359+0000","last_clean":"2024-04-26T20:32:31.366359+0000","last_became_active":"2024-04-26T20:32:26.240012+0000","last_became_peered":"2024-04-26T20:32:26.240012+0000","last_unstale":"2024-04-26T20:32:31.366359+0000","last_undegraded":"2024-04-26T20:32:31.366359+0000","last_fullsized":"2024-04-26T20:32:31.366359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:43:12.776082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550461+0000","last_change":"2024-04-26T20:32:26.224138+0000","last_active":"2024-04-26T20:32:31.550461+0000","last_peered":"2024-04-26T20:32:31.550461+0000","last_clean":"2024-04-26T20:32:31.550461+0000","last_became_active":"2024-04-26T20:32:26.223827+0000","last_became_peered":"2024-04-26T20:32:26.223827+0000","last_unstale":"2024-04-26T20:32:31.550461+0000","last_undegraded":"2024-04-26T20:32:31.550461+0000","last_fullsized":"2024-04-26T20:32:31.550461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:38:33.323001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548862+0000","last_change":"2024-04-26T20:32:26.240298+0000","last_active":"2024-04-26T20:32:31.548862+0000","last_peered":"2024-04-26T20:32:31.548862+0000","last_clean":"2024-04-26T20:32:31.548862+0000","last_became_active":"2024-04-26T20:32:26.239933+0000","last_became_peered":"2024-04-26T20:32:26.239933+0000","last_unstale":"2024-04-26T20:32:31.548862+0000","last_undegraded":"2024-04-26T20:32:31.548862+0000","last_fullsized":"2024-04-26T20:32:31.548862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:00.618136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550850+0000","last_change":"2024-04-26T20:32:26.233053+0000","last_active":"2024-04-26T20:32:31.550850+0000","last_peered":"2024-04-26T20:32:31.550850+0000","last_clean":"2024-04-26T20:32:31.550850+0000","last_became_active":"2024-04-26T20:32:26.232727+0000","last_became_peered":"2024-04-26T20:32:26.232727+0000","last_unstale":"2024-04-26T20:32:31.550850+0000","last_undegraded":"2024-04-26T20:32:31.550850+0000","last_fullsized":"2024-04-26T20:32:31.550850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:35:41.120741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.360730+0000","last_change":"2024-04-26T20:32:26.232600+0000","last_active":"2024-04-26T20:32:31.360730+0000","last_peered":"2024-04-26T20:32:31.360730+0000","last_clean":"2024-04-26T20:32:31.360730+0000","last_became_active":"2024-04-26T20:32:26.232320+0000","last_became_peered":"2024-04-26T20:32:26.232320+0000","last_unstale":"2024-04-26T20:32:31.360730+0000","last_undegraded":"2024-04-26T20:32:31.360730+0000","last_fullsized":"2024-04-26T20:32:31.360730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:18.199940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.362800+0000","last_change":"2024-04-26T20:32:26.215513+0000","last_active":"2024-04-26T20:32:31.362800+0000","last_peered":"2024-04-26T20:32:31.362800+0000","last_clean":"2024-04-26T20:32:31.362800+0000","last_became_active":"2024-04-26T20:32:26.215110+0000","last_became_peered":"2024-04-26T20:32:26.215110+0000","last_unstale":"2024-04-26T20:32:31.362800+0000","last_undegraded":"2024-04-26T20:32:31.362800+0000","last_fullsized":"2024-04-26T20:32:31.362800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:11.356841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548695+0000","last_change":"2024-04-26T20:32:26.230224+0000","last_active":"2024-04-26T20:32:31.548695+0000","last_peered":"2024-04-26T20:32:31.548695+0000","last_clean":"2024-04-26T20:32:31.548695+0000","last_became_active":"2024-04-26T20:32:26.229865+0000","last_became_peered":"2024-04-26T20:32:26.229865+0000","last_unstale":"2024-04-26T20:32:31.548695+0000","last_undegraded":"2024-04-26T20:32:31.548695+0000","last_fullsized":"2024-04-26T20:32:31.548695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:03.060925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551132+0000","last_change":"2024-04-26T20:32:26.219841+0000","last_active":"2024-04-26T20:32:31.551132+0000","last_peered":"2024-04-26T20:32:31.551132+0000","last_clean":"2024-04-26T20:32:31.551132+0000","last_became_active":"2024-04-26T20:32:26.219505+0000","last_became_peered":"2024-04-26T20:32:26.219505+0000","last_unstale":"2024-04-26T20:32:31.551132+0000","last_undegraded":"2024-04-26T20:32:31.551132+0000","last_fullsized":"2024-04-26T20:32:31.551132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:11.514510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361472+0000","last_change":"2024-04-26T20:32:26.226427+0000","last_active":"2024-04-26T20:32:31.361472+0000","last_peered":"2024-04-26T20:32:31.361472+0000","last_clean":"2024-04-26T20:32:31.361472+0000","last_became_active":"2024-04-26T20:32:26.226112+0000","last_became_peered":"2024-04-26T20:32:26.226112+0000","last_unstale":"2024-04-26T20:32:31.361472+0000","last_undegraded":"2024-04-26T20:32:31.361472+0000","last_fullsized":"2024-04-26T20:32:31.361472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:56.506369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550356+0000","last_change":"2024-04-26T20:32:26.222756+0000","last_active":"2024-04-26T20:32:31.550356+0000","last_peered":"2024-04-26T20:32:31.550356+0000","last_clean":"2024-04-26T20:32:31.550356+0000","last_became_active":"2024-04-26T20:32:26.222342+0000","last_became_peered":"2024-04-26T20:32:26.222342+0000","last_unstale":"2024-04-26T20:32:31.550356+0000","last_undegraded":"2024-04-26T20:32:31.550356+0000","last_fullsized":"2024-04-26T20:32:31.550356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:11.543434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.552125+0000","last_change":"2024-04-26T20:32:26.242864+0000","last_active":"2024-04-26T20:32:31.552125+0000","last_peered":"2024-04-26T20:32:31.552125+0000","last_clean":"2024-04-26T20:32:31.552125+0000","last_became_active":"2024-04-26T20:32:26.242511+0000","last_became_peered":"2024-04-26T20:32:26.242511+0000","last_unstale":"2024-04-26T20:32:31.552125+0000","last_undegraded":"2024-04-26T20:32:31.552125+0000","last_fullsized":"2024-04-26T20:32:31.552125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:26:37.412901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.365379+0000","last_change":"2024-04-26T20:32:26.247783+0000","last_active":"2024-04-26T20:32:31.365379+0000","last_peered":"2024-04-26T20:32:31.365379+0000","last_clean":"2024-04-26T20:32:31.365379+0000","last_became_active":"2024-04-26T20:32:26.247466+0000","last_became_peered":"2024-04-26T20:32:26.247466+0000","last_unstale":"2024-04-26T20:32:31.365379+0000","last_undegraded":"2024-04-26T20:32:31.365379+0000","last_fullsized":"2024-04-26T20:32:31.365379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:36:59.424377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.364729+0000","last_change":"2024-04-26T20:32:26.249823+0000","last_active":"2024-04-26T20:32:31.364729+0000","last_peered":"2024-04-26T20:32:31.364729+0000","last_clean":"2024-04-26T20:32:31.364729+0000","last_became_active":"2024-04-26T20:32:26.248954+0000","last_became_peered":"2024-04-26T20:32:26.248954+0000","last_unstale":"2024-04-26T20:32:31.364729+0000","last_undegraded":"2024-04-26T20:32:31.364729+0000","last_fullsized":"2024-04-26T20:32:31.364729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:55:44.246537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548108+0000","last_change":"2024-04-26T20:32:26.241997+0000","last_active":"2024-04-26T20:32:31.548108+0000","last_peered":"2024-04-26T20:32:31.548108+0000","last_clean":"2024-04-26T20:32:31.548108+0000","last_became_active":"2024-04-26T20:32:26.240856+0000","last_became_peered":"2024-04-26T20:32:26.240856+0000","last_unstale":"2024-04-26T20:32:31.548108+0000","last_undegraded":"2024-04-26T20:32:31.548108+0000","last_fullsized":"2024-04-26T20:32:31.548108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:53.895994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551455+0000","last_change":"2024-04-26T20:32:26.241097+0000","last_active":"2024-04-26T20:32:31.551455+0000","last_peered":"2024-04-26T20:32:31.551455+0000","last_clean":"2024-04-26T20:32:31.551455+0000","last_became_active":"2024-04-26T20:32:26.240770+0000","last_became_peered":"2024-04-26T20:32:26.240770+0000","last_unstale":"2024-04-26T20:32:31.551455+0000","last_undegraded":"2024-04-26T20:32:31.551455+0000","last_fullsized":"2024-04-26T20:32:31.551455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:10.443314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551349+0000","last_change":"2024-04-26T20:32:26.232279+0000","last_active":"2024-04-26T20:32:31.551349+0000","last_peered":"2024-04-26T20:32:31.551349+0000","last_clean":"2024-04-26T20:32:31.551349+0000","last_became_active":"2024-04-26T20:32:26.231256+0000","last_became_peered":"2024-04-26T20:32:26.231256+0000","last_unstale":"2024-04-26T20:32:31.551349+0000","last_undegraded":"2024-04-26T20:32:31.551349+0000","last_fullsized":"2024-04-26T20:32:31.551349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:26.484932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550404+0000","last_change":"2024-04-26T20:32:26.249876+0000","last_active":"2024-04-26T20:32:31.550404+0000","last_peered":"2024-04-26T20:32:31.550404+0000","last_clean":"2024-04-26T20:32:31.550404+0000","last_became_active":"2024-04-26T20:32:26.249416+0000","last_became_peered":"2024-04-26T20:32:26.249416+0000","last_unstale":"2024-04-26T20:32:31.550404+0000","last_undegraded":"2024-04-26T20:32:31.550404+0000","last_fullsized":"2024-04-26T20:32:31.550404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:54.616941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549613+0000","last_change":"2024-04-26T20:32:30.389128+0000","last_active":"2024-04-26T20:32:31.549613+0000","last_peered":"2024-04-26T20:32:31.549613+0000","last_clean":"2024-04-26T20:32:31.549613+0000","last_became_active":"2024-04-26T20:32:30.388668+0000","last_became_peered":"2024-04-26T20:32:30.388668+0000","last_unstale":"2024-04-26T20:32:31.549613+0000","last_undegraded":"2024-04-26T20:32:31.549613+0000","last_fullsized":"2024-04-26T20:32:31.549613+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:12.019691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:54.687894+0000","last_change":"2024-04-26T20:32:25.278417+0000","last_active":"2024-04-26T20:32:54.687894+0000","last_peered":"2024-04-26T20:32:54.687894+0000","last_clean":"2024-04-26T20:32:54.687894+0000","last_became_active":"2024-04-26T20:32:25.278004+0000","last_became_peered":"2024-04-26T20:32:25.278004+0000","last_unstale":"2024-04-26T20:32:54.687894+0000","last_undegraded":"2024-04-26T20:32:54.687894+0000","last_fullsized":"2024-04-26T20:32:54.687894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:51.458285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.365337+0000","last_change":"2024-04-26T20:32:26.229015+0000","last_active":"2024-04-26T20:32:31.365337+0000","last_peered":"2024-04-26T20:32:31.365337+0000","last_clean":"2024-04-26T20:32:31.365337+0000","last_became_active":"2024-04-26T20:32:26.228729+0000","last_became_peered":"2024-04-26T20:32:26.228729+0000","last_unstale":"2024-04-26T20:32:31.365337+0000","last_undegraded":"2024-04-26T20:32:31.365337+0000","last_fullsized":"2024-04-26T20:32:31.365337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:20.014133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549341+0000","last_change":"2024-04-26T20:32:30.383196+0000","last_active":"2024-04-26T20:32:31.549341+0000","last_peered":"2024-04-26T20:32:31.549341+0000","last_clean":"2024-04-26T20:32:31.549341+0000","last_became_active":"2024-04-26T20:32:30.382831+0000","last_became_peered":"2024-04-26T20:32:30.382831+0000","last_unstale":"2024-04-26T20:32:31.549341+0000","last_undegraded":"2024-04-26T20:32:31.549341+0000","last_fullsized":"2024-04-26T20:32:31.549341+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:48:13.696394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366869+0000","last_change":"2024-04-26T20:32:25.294227+0000","last_active":"2024-04-26T20:32:31.366869+0000","last_peered":"2024-04-26T20:32:31.366869+0000","last_clean":"2024-04-26T20:32:31.366869+0000","last_became_active":"2024-04-26T20:32:25.287681+0000","last_became_peered":"2024-04-26T20:32:25.287681+0000","last_unstale":"2024-04-26T20:32:31.366869+0000","last_undegraded":"2024-04-26T20:32:31.366869+0000","last_fullsized":"2024-04-26T20:32:31.366869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:25.567135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.368440+0000","last_change":"2024-04-26T20:32:26.251364+0000","last_active":"2024-04-26T20:32:31.368440+0000","last_peered":"2024-04-26T20:32:31.368440+0000","last_clean":"2024-04-26T20:32:31.368440+0000","last_became_active":"2024-04-26T20:32:26.251037+0000","last_became_peered":"2024-04-26T20:32:26.251037+0000","last_unstale":"2024-04-26T20:32:31.368440+0000","last_undegraded":"2024-04-26T20:32:31.368440+0000","last_fullsized":"2024-04-26T20:32:31.368440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:17:41.813889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550321+0000","last_change":"2024-04-26T20:32:30.385134+0000","last_active":"2024-04-26T20:32:31.550321+0000","last_peered":"2024-04-26T20:32:31.550321+0000","last_clean":"2024-04-26T20:32:31.550321+0000","last_became_active":"2024-04-26T20:32:30.384694+0000","last_became_peered":"2024-04-26T20:32:30.384694+0000","last_unstale":"2024-04-26T20:32:31.550321+0000","last_undegraded":"2024-04-26T20:32:31.550321+0000","last_fullsized":"2024-04-26T20:32:31.550321+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:59.622623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'11","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:54.686164+0000","last_change":"2024-04-26T20:32:25.267156+0000","last_active":"2024-04-26T20:32:54.686164+0000","last_peered":"2024-04-26T20:32:54.686164+0000","last_clean":"2024-04-26T20:32:54.686164+0000","last_became_active":"2024-04-26T20:32:25.266113+0000","last_became_peered":"2024-04-26T20:32:25.266113+0000","last_unstale":"2024-04-26T20:32:54.686164+0000","last_undegraded":"2024-04-26T20:32:54.686164+0000","last_fullsized":"2024-04-26T20:32:54.686164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:30:38.798437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548869+0000","last_change":"2024-04-26T20:32:26.249908+0000","last_active":"2024-04-26T20:32:31.548869+0000","last_peered":"2024-04-26T20:32:31.548869+0000","last_clean":"2024-04-26T20:32:31.548869+0000","last_became_active":"2024-04-26T20:32:26.249576+0000","last_became_peered":"2024-04-26T20:32:26.249576+0000","last_unstale":"2024-04-26T20:32:31.548869+0000","last_undegraded":"2024-04-26T20:32:31.548869+0000","last_fullsized":"2024-04-26T20:32:31.548869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:37:09.611624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549802+0000","last_change":"2024-04-26T20:32:30.391456+0000","last_active":"2024-04-26T20:32:31.549802+0000","last_peered":"2024-04-26T20:32:31.549802+0000","last_clean":"2024-04-26T20:32:31.549802+0000","last_became_active":"2024-04-26T20:32:30.391028+0000","last_became_peered":"2024-04-26T20:32:30.391028+0000","last_unstale":"2024-04-26T20:32:31.549802+0000","last_undegraded":"2024-04-26T20:32:31.549802+0000","last_fullsized":"2024-04-26T20:32:31.549802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:55:17.987421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:54.685217+0000","last_change":"2024-04-26T20:32:25.264519+0000","last_active":"2024-04-26T20:32:54.685217+0000","last_peered":"2024-04-26T20:32:54.685217+0000","last_clean":"2024-04-26T20:32:54.685217+0000","last_became_active":"2024-04-26T20:32:25.264189+0000","last_became_peered":"2024-04-26T20:32:25.264189+0000","last_unstale":"2024-04-26T20:32:54.685217+0000","last_undegraded":"2024-04-26T20:32:54.685217+0000","last_fullsized":"2024-04-26T20:32:54.685217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:12:56.167119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551106+0000","last_change":"2024-04-26T20:32:26.226281+0000","last_active":"2024-04-26T20:32:31.551106+0000","last_peered":"2024-04-26T20:32:31.551106+0000","last_clean":"2024-04-26T20:32:31.551106+0000","last_became_active":"2024-04-26T20:32:26.225958+0000","last_became_peered":"2024-04-26T20:32:26.225958+0000","last_unstale":"2024-04-26T20:32:31.551106+0000","last_undegraded":"2024-04-26T20:32:31.551106+0000","last_fullsized":"2024-04-26T20:32:31.551106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:22:30.914293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550306+0000","last_change":"2024-04-26T20:32:30.379734+0000","last_active":"2024-04-26T20:32:31.550306+0000","last_peered":"2024-04-26T20:32:31.550306+0000","last_clean":"2024-04-26T20:32:31.550306+0000","last_became_active":"2024-04-26T20:32:30.379240+0000","last_became_peered":"2024-04-26T20:32:30.379240+0000","last_unstale":"2024-04-26T20:32:31.550306+0000","last_undegraded":"2024-04-26T20:32:31.550306+0000","last_fullsized":"2024-04-26T20:32:31.550306+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:05:46.669583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.657470+0000","last_change":"2024-04-26T20:32:25.268389+0000","last_active":"2024-04-26T20:32:51.657470+0000","last_peered":"2024-04-26T20:32:51.657470+0000","last_clean":"2024-04-26T20:32:51.657470+0000","last_became_active":"2024-04-26T20:32:25.268060+0000","last_became_peered":"2024-04-26T20:32:25.268060+0000","last_unstale":"2024-04-26T20:32:51.657470+0000","last_undegraded":"2024-04-26T20:32:51.657470+0000","last_fullsized":"2024-04-26T20:32:51.657470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:38.636609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548835+0000","last_change":"2024-04-26T20:32:26.235743+0000","last_active":"2024-04-26T20:32:31.548835+0000","last_peered":"2024-04-26T20:32:31.548835+0000","last_clean":"2024-04-26T20:32:31.548835+0000","last_became_active":"2024-04-26T20:32:26.235375+0000","last_became_peered":"2024-04-26T20:32:26.235375+0000","last_unstale":"2024-04-26T20:32:31.548835+0000","last_undegraded":"2024-04-26T20:32:31.548835+0000","last_fullsized":"2024-04-26T20:32:31.548835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:27.673837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550310+0000","last_change":"2024-04-26T20:32:30.390260+0000","last_active":"2024-04-26T20:32:31.550310+0000","last_peered":"2024-04-26T20:32:31.550310+0000","last_clean":"2024-04-26T20:32:31.550310+0000","last_became_active":"2024-04-26T20:32:30.389796+0000","last_became_peered":"2024-04-26T20:32:30.389796+0000","last_unstale":"2024-04-26T20:32:31.550310+0000","last_undegraded":"2024-04-26T20:32:31.550310+0000","last_fullsized":"2024-04-26T20:32:31.550310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:16:13.399131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:52.667686+0000","last_change":"2024-04-26T20:32:25.301063+0000","last_active":"2024-04-26T20:32:52.667686+0000","last_peered":"2024-04-26T20:32:52.667686+0000","last_clean":"2024-04-26T20:32:52.667686+0000","last_became_active":"2024-04-26T20:32:25.300755+0000","last_became_peered":"2024-04-26T20:32:25.300755+0000","last_unstale":"2024-04-26T20:32:52.667686+0000","last_undegraded":"2024-04-26T20:32:52.667686+0000","last_fullsized":"2024-04-26T20:32:52.667686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:50.542961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550467+0000","last_change":"2024-04-26T20:32:26.223507+0000","last_active":"2024-04-26T20:32:31.550467+0000","last_peered":"2024-04-26T20:32:31.550467+0000","last_clean":"2024-04-26T20:32:31.550467+0000","last_became_active":"2024-04-26T20:32:26.222991+0000","last_became_peered":"2024-04-26T20:32:26.222991+0000","last_unstale":"2024-04-26T20:32:31.550467+0000","last_undegraded":"2024-04-26T20:32:31.550467+0000","last_fullsized":"2024-04-26T20:32:31.550467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:53:25.207884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551519+0000","last_change":"2024-04-26T20:32:30.389354+0000","last_active":"2024-04-26T20:32:31.551519+0000","last_peered":"2024-04-26T20:32:31.551519+0000","last_clean":"2024-04-26T20:32:31.551519+0000","last_became_active":"2024-04-26T20:32:30.388848+0000","last_became_peered":"2024-04-26T20:32:30.388848+0000","last_unstale":"2024-04-26T20:32:31.551519+0000","last_undegraded":"2024-04-26T20:32:31.551519+0000","last_fullsized":"2024-04-26T20:32:31.551519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:12.615750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:52.669205+0000","last_change":"2024-04-26T20:32:25.292694+0000","last_active":"2024-04-26T20:32:52.669205+0000","last_peered":"2024-04-26T20:32:52.669205+0000","last_clean":"2024-04-26T20:32:52.669205+0000","last_became_active":"2024-04-26T20:32:25.283757+0000","last_became_peered":"2024-04-26T20:32:25.283757+0000","last_unstale":"2024-04-26T20:32:52.669205+0000","last_undegraded":"2024-04-26T20:32:52.669205+0000","last_fullsized":"2024-04-26T20:32:52.669205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:05:15.294842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.363566+0000","last_change":"2024-04-26T20:32:26.232770+0000","last_active":"2024-04-26T20:32:31.363566+0000","last_peered":"2024-04-26T20:32:31.363566+0000","last_clean":"2024-04-26T20:32:31.363566+0000","last_became_active":"2024-04-26T20:32:26.232381+0000","last_became_peered":"2024-04-26T20:32:26.232381+0000","last_unstale":"2024-04-26T20:32:31.363566+0000","last_undegraded":"2024-04-26T20:32:31.363566+0000","last_fullsized":"2024-04-26T20:32:31.363566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:16:43.310582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366947+0000","last_change":"2024-04-26T20:32:30.389690+0000","last_active":"2024-04-26T20:32:31.366947+0000","last_peered":"2024-04-26T20:32:31.366947+0000","last_clean":"2024-04-26T20:32:31.366947+0000","last_became_active":"2024-04-26T20:32:30.389258+0000","last_became_peered":"2024-04-26T20:32:30.389258+0000","last_unstale":"2024-04-26T20:32:31.366947+0000","last_undegraded":"2024-04-26T20:32:31.366947+0000","last_fullsized":"2024-04-26T20:32:31.366947+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:36.528757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.656122+0000","last_change":"2024-04-26T20:32:25.267681+0000","last_active":"2024-04-26T20:32:51.656122+0000","last_peered":"2024-04-26T20:32:51.656122+0000","last_clean":"2024-04-26T20:32:51.656122+0000","last_became_active":"2024-04-26T20:32:25.267351+0000","last_became_peered":"2024-04-26T20:32:25.267351+0000","last_unstale":"2024-04-26T20:32:51.656122+0000","last_undegraded":"2024-04-26T20:32:51.656122+0000","last_fullsized":"2024-04-26T20:32:51.656122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:00:26.339458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550206+0000","last_change":"2024-04-26T20:32:26.243638+0000","last_active":"2024-04-26T20:32:31.550206+0000","last_peered":"2024-04-26T20:32:31.550206+0000","last_clean":"2024-04-26T20:32:31.550206+0000","last_became_active":"2024-04-26T20:32:26.241783+0000","last_became_peered":"2024-04-26T20:32:26.241783+0000","last_unstale":"2024-04-26T20:32:31.550206+0000","last_undegraded":"2024-04-26T20:32:31.550206+0000","last_fullsized":"2024-04-26T20:32:31.550206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:20.821967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.364193+0000","last_change":"2024-04-26T20:32:30.382937+0000","last_active":"2024-04-26T20:32:31.364193+0000","last_peered":"2024-04-26T20:32:31.364193+0000","last_clean":"2024-04-26T20:32:31.364193+0000","last_became_active":"2024-04-26T20:32:30.378828+0000","last_became_peered":"2024-04-26T20:32:30.378828+0000","last_unstale":"2024-04-26T20:32:31.364193+0000","last_undegraded":"2024-04-26T20:32:31.364193+0000","last_fullsized":"2024-04-26T20:32:31.364193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:33:41.685012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.656472+0000","last_change":"2024-04-26T20:32:25.280362+0000","last_active":"2024-04-26T20:32:51.656472+0000","last_peered":"2024-04-26T20:32:51.656472+0000","last_clean":"2024-04-26T20:32:51.656472+0000","last_became_active":"2024-04-26T20:32:25.279975+0000","last_became_peered":"2024-04-26T20:32:25.279975+0000","last_unstale":"2024-04-26T20:32:51.656472+0000","last_undegraded":"2024-04-26T20:32:51.656472+0000","last_fullsized":"2024-04-26T20:32:51.656472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:08.150635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548433+0000","last_change":"2024-04-26T20:32:26.231231+0000","last_active":"2024-04-26T20:32:31.548433+0000","last_peered":"2024-04-26T20:32:31.548433+0000","last_clean":"2024-04-26T20:32:31.548433+0000","last_became_active":"2024-04-26T20:32:26.230831+0000","last_became_peered":"2024-04-26T20:32:26.230831+0000","last_unstale":"2024-04-26T20:32:31.548433+0000","last_undegraded":"2024-04-26T20:32:31.548433+0000","last_fullsized":"2024-04-26T20:32:31.548433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:03:45.272542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549920+0000","last_change":"2024-04-26T20:32:30.382633+0000","last_active":"2024-04-26T20:32:31.549920+0000","last_peered":"2024-04-26T20:32:31.549920+0000","last_clean":"2024-04-26T20:32:31.549920+0000","last_became_active":"2024-04-26T20:32:30.379821+0000","last_became_peered":"2024-04-26T20:32:30.379821+0000","last_unstale":"2024-04-26T20:32:31.549920+0000","last_undegraded":"2024-04-26T20:32:31.549920+0000","last_fullsized":"2024-04-26T20:32:31.549920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:58:16.888172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.652483+0000","last_change":"2024-04-26T20:32:25.264844+0000","last_active":"2024-04-26T20:32:51.652483+0000","last_peered":"2024-04-26T20:32:51.652483+0000","last_clean":"2024-04-26T20:32:51.652483+0000","last_became_active":"2024-04-26T20:32:25.264097+0000","last_became_peered":"2024-04-26T20:32:25.264097+0000","last_unstale":"2024-04-26T20:32:51.652483+0000","last_undegraded":"2024-04-26T20:32:51.652483+0000","last_fullsized":"2024-04-26T20:32:51.652483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:09.308656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.368581+0000","last_change":"2024-04-26T20:32:26.248211+0000","last_active":"2024-04-26T20:32:31.368581+0000","last_peered":"2024-04-26T20:32:31.368581+0000","last_clean":"2024-04-26T20:32:31.368581+0000","last_became_active":"2024-04-26T20:32:26.247812+0000","last_became_peered":"2024-04-26T20:32:26.247812+0000","last_unstale":"2024-04-26T20:32:31.368581+0000","last_undegraded":"2024-04-26T20:32:31.368581+0000","last_fullsized":"2024-04-26T20:32:31.368581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:59:05.448957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550692+0000","last_change":"2024-04-26T20:32:30.377547+0000","last_active":"2024-04-26T20:32:31.550692+0000","last_peered":"2024-04-26T20:32:31.550692+0000","last_clean":"2024-04-26T20:32:31.550692+0000","last_became_active":"2024-04-26T20:32:30.377139+0000","last_became_peered":"2024-04-26T20:32:30.377139+0000","last_unstale":"2024-04-26T20:32:31.550692+0000","last_undegraded":"2024-04-26T20:32:31.550692+0000","last_fullsized":"2024-04-26T20:32:31.550692+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:41.970710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.655702+0000","last_change":"2024-04-26T20:32:25.284996+0000","last_active":"2024-04-26T20:32:51.655702+0000","last_peered":"2024-04-26T20:32:51.655702+0000","last_clean":"2024-04-26T20:32:51.655702+0000","last_became_active":"2024-04-26T20:32:25.284702+0000","last_became_peered":"2024-04-26T20:32:25.284702+0000","last_unstale":"2024-04-26T20:32:51.655702+0000","last_undegraded":"2024-04-26T20:32:51.655702+0000","last_fullsized":"2024-04-26T20:32:51.655702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:39.831522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548777+0000","last_change":"2024-04-26T20:32:26.230357+0000","last_active":"2024-04-26T20:32:31.548777+0000","last_peered":"2024-04-26T20:32:31.548777+0000","last_clean":"2024-04-26T20:32:31.548777+0000","last_became_active":"2024-04-26T20:32:26.229831+0000","last_became_peered":"2024-04-26T20:32:26.229831+0000","last_unstale":"2024-04-26T20:32:31.548777+0000","last_undegraded":"2024-04-26T20:32:31.548777+0000","last_fullsized":"2024-04-26T20:32:31.548777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:20:12.452768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550597+0000","last_change":"2024-04-26T20:32:30.383469+0000","last_active":"2024-04-26T20:32:31.550597+0000","last_peered":"2024-04-26T20:32:31.550597+0000","last_clean":"2024-04-26T20:32:31.550597+0000","last_became_active":"2024-04-26T20:32:30.382416+0000","last_became_peered":"2024-04-26T20:32:30.382416+0000","last_unstale":"2024-04-26T20:32:31.550597+0000","last_undegraded":"2024-04-26T20:32:31.550597+0000","last_fullsized":"2024-04-26T20:32:31.550597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:55.254214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.654974+0000","last_change":"2024-04-26T20:32:25.264433+0000","last_active":"2024-04-26T20:32:51.654974+0000","last_peered":"2024-04-26T20:32:51.654974+0000","last_clean":"2024-04-26T20:32:51.654974+0000","last_became_active":"2024-04-26T20:32:25.264104+0000","last_became_peered":"2024-04-26T20:32:25.264104+0000","last_unstale":"2024-04-26T20:32:51.654974+0000","last_undegraded":"2024-04-26T20:32:51.654974+0000","last_fullsized":"2024-04-26T20:32:51.654974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.030082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549425+0000","last_change":"2024-04-26T20:32:26.219704+0000","last_active":"2024-04-26T20:32:31.549425+0000","last_peered":"2024-04-26T20:32:31.549425+0000","last_clean":"2024-04-26T20:32:31.549425+0000","last_became_active":"2024-04-26T20:32:26.219397+0000","last_became_peered":"2024-04-26T20:32:26.219397+0000","last_unstale":"2024-04-26T20:32:31.549425+0000","last_undegraded":"2024-04-26T20:32:31.549425+0000","last_fullsized":"2024-04-26T20:32:31.549425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:04.570044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551493+0000","last_change":"2024-04-26T20:32:30.388828+0000","last_active":"2024-04-26T20:32:31.551493+0000","last_peered":"2024-04-26T20:32:31.551493+0000","last_clean":"2024-04-26T20:32:31.551493+0000","last_became_active":"2024-04-26T20:32:30.387748+0000","last_became_peered":"2024-04-26T20:32:30.387748+0000","last_unstale":"2024-04-26T20:32:31.551493+0000","last_undegraded":"2024-04-26T20:32:31.551493+0000","last_fullsized":"2024-04-26T20:32:31.551493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:53:09.900151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550046+0000","last_change":"2024-04-26T20:32:25.271910+0000","last_active":"2024-04-26T20:32:31.550046+0000","last_peered":"2024-04-26T20:32:31.550046+0000","last_clean":"2024-04-26T20:32:31.550046+0000","last_became_active":"2024-04-26T20:32:25.271572+0000","last_became_peered":"2024-04-26T20:32:25.271572+0000","last_unstale":"2024-04-26T20:32:31.550046+0000","last_undegraded":"2024-04-26T20:32:31.550046+0000","last_fullsized":"2024-04-26T20:32:31.550046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:32:59.013229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551909+0000","last_change":"2024-04-26T20:32:26.237691+0000","last_active":"2024-04-26T20:32:31.551909+0000","last_peered":"2024-04-26T20:32:31.551909+0000","last_clean":"2024-04-26T20:32:31.551909+0000","last_became_active":"2024-04-26T20:32:26.237338+0000","last_became_peered":"2024-04-26T20:32:26.237338+0000","last_unstale":"2024-04-26T20:32:31.551909+0000","last_undegraded":"2024-04-26T20:32:31.551909+0000","last_fullsized":"2024-04-26T20:32:31.551909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:51.782689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551138+0000","last_change":"2024-04-26T20:32:30.388485+0000","last_active":"2024-04-26T20:32:31.551138+0000","last_peered":"2024-04-26T20:32:31.551138+0000","last_clean":"2024-04-26T20:32:31.551138+0000","last_became_active":"2024-04-26T20:32:30.388074+0000","last_became_peered":"2024-04-26T20:32:30.388074+0000","last_unstale":"2024-04-26T20:32:31.551138+0000","last_undegraded":"2024-04-26T20:32:31.551138+0000","last_fullsized":"2024-04-26T20:32:31.551138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:52.792551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.658845+0000","last_change":"2024-04-26T20:32:25.271589+0000","last_active":"2024-04-26T20:32:51.658845+0000","last_peered":"2024-04-26T20:32:51.658845+0000","last_clean":"2024-04-26T20:32:51.658845+0000","last_became_active":"2024-04-26T20:32:25.271260+0000","last_became_peered":"2024-04-26T20:32:25.271260+0000","last_unstale":"2024-04-26T20:32:51.658845+0000","last_undegraded":"2024-04-26T20:32:51.658845+0000","last_fullsized":"2024-04-26T20:32:51.658845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:30.978561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361096+0000","last_change":"2024-04-26T20:32:26.240158+0000","last_active":"2024-04-26T20:32:31.361096+0000","last_peered":"2024-04-26T20:32:31.361096+0000","last_clean":"2024-04-26T20:32:31.361096+0000","last_became_active":"2024-04-26T20:32:26.239831+0000","last_became_peered":"2024-04-26T20:32:26.239831+0000","last_unstale":"2024-04-26T20:32:31.361096+0000","last_undegraded":"2024-04-26T20:32:31.361096+0000","last_fullsized":"2024-04-26T20:32:31.361096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:07.870097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551073+0000","last_change":"2024-04-26T20:32:30.392090+0000","last_active":"2024-04-26T20:32:31.551073+0000","last_peered":"2024-04-26T20:32:31.551073+0000","last_clean":"2024-04-26T20:32:31.551073+0000","last_became_active":"2024-04-26T20:32:30.390404+0000","last_became_peered":"2024-04-26T20:32:30.390404+0000","last_unstale":"2024-04-26T20:32:31.551073+0000","last_undegraded":"2024-04-26T20:32:31.551073+0000","last_fullsized":"2024-04-26T20:32:31.551073+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:50:10.743133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:53.674608+0000","last_change":"2024-04-26T20:32:25.264291+0000","last_active":"2024-04-26T20:32:53.674608+0000","last_peered":"2024-04-26T20:32:53.674608+0000","last_clean":"2024-04-26T20:32:53.674608+0000","last_became_active":"2024-04-26T20:32:25.263901+0000","last_became_peered":"2024-04-26T20:32:25.263901+0000","last_unstale":"2024-04-26T20:32:53.674608+0000","last_undegraded":"2024-04-26T20:32:53.674608+0000","last_fullsized":"2024-04-26T20:32:53.674608+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:23:28.851941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.360941+0000","last_change":"2024-04-26T20:32:26.221472+0000","last_active":"2024-04-26T20:32:31.360941+0000","last_peered":"2024-04-26T20:32:31.360941+0000","last_clean":"2024-04-26T20:32:31.360941+0000","last_became_active":"2024-04-26T20:32:26.221117+0000","last_became_peered":"2024-04-26T20:32:26.221117+0000","last_unstale":"2024-04-26T20:32:31.360941+0000","last_undegraded":"2024-04-26T20:32:31.360941+0000","last_fullsized":"2024-04-26T20:32:31.360941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:27:01.666749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.553818+0000","last_change":"2024-04-26T20:32:30.392000+0000","last_active":"2024-04-26T20:32:31.553818+0000","last_peered":"2024-04-26T20:32:31.553818+0000","last_clean":"2024-04-26T20:32:31.553818+0000","last_became_active":"2024-04-26T20:32:30.386933+0000","last_became_peered":"2024-04-26T20:32:30.386933+0000","last_unstale":"2024-04-26T20:32:31.553818+0000","last_undegraded":"2024-04-26T20:32:31.553818+0000","last_fullsized":"2024-04-26T20:32:31.553818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:23.666439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:53.679043+0000","last_change":"2024-04-26T20:32:25.276482+0000","last_active":"2024-04-26T20:32:53.679043+0000","last_peered":"2024-04-26T20:32:53.679043+0000","last_clean":"2024-04-26T20:32:53.679043+0000","last_became_active":"2024-04-26T20:32:25.276159+0000","last_became_peered":"2024-04-26T20:32:25.276159+0000","last_unstale":"2024-04-26T20:32:53.679043+0000","last_undegraded":"2024-04-26T20:32:53.679043+0000","last_fullsized":"2024-04-26T20:32:53.679043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:34.029524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549686+0000","last_change":"2024-04-26T20:32:26.225344+0000","last_active":"2024-04-26T20:32:31.549686+0000","last_peered":"2024-04-26T20:32:31.549686+0000","last_clean":"2024-04-26T20:32:31.549686+0000","last_became_active":"2024-04-26T20:32:26.225039+0000","last_became_peered":"2024-04-26T20:32:26.225039+0000","last_unstale":"2024-04-26T20:32:31.549686+0000","last_undegraded":"2024-04-26T20:32:31.549686+0000","last_fullsized":"2024-04-26T20:32:31.549686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:08.680765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549624+0000","last_change":"2024-04-26T20:32:30.389274+0000","last_active":"2024-04-26T20:32:31.549624+0000","last_peered":"2024-04-26T20:32:31.549624+0000","last_clean":"2024-04-26T20:32:31.549624+0000","last_became_active":"2024-04-26T20:32:30.388527+0000","last_became_peered":"2024-04-26T20:32:30.388527+0000","last_unstale":"2024-04-26T20:32:31.549624+0000","last_undegraded":"2024-04-26T20:32:31.549624+0000","last_fullsized":"2024-04-26T20:32:31.549624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:14.689223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549588+0000","last_change":"2024-04-26T20:32:25.267338+0000","last_active":"2024-04-26T20:32:31.549588+0000","last_peered":"2024-04-26T20:32:31.549588+0000","last_clean":"2024-04-26T20:32:31.549588+0000","last_became_active":"2024-04-26T20:32:25.266234+0000","last_became_peered":"2024-04-26T20:32:25.266234+0000","last_unstale":"2024-04-26T20:32:31.549588+0000","last_undegraded":"2024-04-26T20:32:31.549588+0000","last_fullsized":"2024-04-26T20:32:31.549588+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:55.603147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551257+0000","last_change":"2024-04-26T20:32:26.222686+0000","last_active":"2024-04-26T20:32:31.551257+0000","last_peered":"2024-04-26T20:32:31.551257+0000","last_clean":"2024-04-26T20:32:31.551257+0000","last_became_active":"2024-04-26T20:32:26.222364+0000","last_became_peered":"2024-04-26T20:32:26.222364+0000","last_unstale":"2024-04-26T20:32:31.551257+0000","last_undegraded":"2024-04-26T20:32:31.551257+0000","last_fullsized":"2024-04-26T20:32:31.551257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:56.170913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548918+0000","last_change":"2024-04-26T20:32:30.414367+0000","last_active":"2024-04-26T20:32:31.548918+0000","last_peered":"2024-04-26T20:32:31.548918+0000","last_clean":"2024-04-26T20:32:31.548918+0000","last_became_active":"2024-04-26T20:32:30.413889+0000","last_became_peered":"2024-04-26T20:32:30.413889+0000","last_unstale":"2024-04-26T20:32:31.548918+0000","last_undegraded":"2024-04-26T20:32:31.548918+0000","last_fullsized":"2024-04-26T20:32:31.548918+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:11.336500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550321+0000","last_change":"2024-04-26T20:32:25.286408+0000","last_active":"2024-04-26T20:32:31.550321+0000","last_peered":"2024-04-26T20:32:31.550321+0000","last_clean":"2024-04-26T20:32:31.550321+0000","last_became_active":"2024-04-26T20:32:25.285986+0000","last_became_peered":"2024-04-26T20:32:25.285986+0000","last_unstale":"2024-04-26T20:32:31.550321+0000","last_undegraded":"2024-04-26T20:32:31.550321+0000","last_fullsized":"2024-04-26T20:32:31.550321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:47.757720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550746+0000","last_change":"2024-04-26T20:32:26.250459+0000","last_active":"2024-04-26T20:32:31.550746+0000","last_peered":"2024-04-26T20:32:31.550746+0000","last_clean":"2024-04-26T20:32:31.550746+0000","last_became_active":"2024-04-26T20:32:26.250154+0000","last_became_peered":"2024-04-26T20:32:26.250154+0000","last_unstale":"2024-04-26T20:32:31.550746+0000","last_undegraded":"2024-04-26T20:32:31.550746+0000","last_fullsized":"2024-04-26T20:32:31.550746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:47.185067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.552702+0000","last_change":"2024-04-26T20:32:30.395258+0000","last_active":"2024-04-26T20:32:31.552702+0000","last_peered":"2024-04-26T20:32:31.552702+0000","last_clean":"2024-04-26T20:32:31.552702+0000","last_became_active":"2024-04-26T20:32:30.392452+0000","last_became_peered":"2024-04-26T20:32:30.392452+0000","last_unstale":"2024-04-26T20:32:31.552702+0000","last_undegraded":"2024-04-26T20:32:31.552702+0000","last_fullsized":"2024-04-26T20:32:31.552702+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:19.883901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361095+0000","last_change":"2024-04-26T20:32:25.297361+0000","last_active":"2024-04-26T20:32:31.361095+0000","last_peered":"2024-04-26T20:32:31.361095+0000","last_clean":"2024-04-26T20:32:31.361095+0000","last_became_active":"2024-04-26T20:32:25.297125+0000","last_became_peered":"2024-04-26T20:32:25.297125+0000","last_unstale":"2024-04-26T20:32:31.361095+0000","last_undegraded":"2024-04-26T20:32:31.361095+0000","last_fullsized":"2024-04-26T20:32:31.361095+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:14:38.038683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550058+0000","last_change":"2024-04-26T20:32:26.249371+0000","last_active":"2024-04-26T20:32:31.550058+0000","last_peered":"2024-04-26T20:32:31.550058+0000","last_clean":"2024-04-26T20:32:31.550058+0000","last_became_active":"2024-04-26T20:32:26.249018+0000","last_became_peered":"2024-04-26T20:32:26.249018+0000","last_unstale":"2024-04-26T20:32:31.550058+0000","last_undegraded":"2024-04-26T20:32:31.550058+0000","last_fullsized":"2024-04-26T20:32:31.550058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:27.553928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.553674+0000","last_change":"2024-04-26T20:32:30.414854+0000","last_active":"2024-04-26T20:32:31.553674+0000","last_peered":"2024-04-26T20:32:31.553674+0000","last_clean":"2024-04-26T20:32:31.553674+0000","last_became_active":"2024-04-26T20:32:30.414374+0000","last_became_peered":"2024-04-26T20:32:30.414374+0000","last_unstale":"2024-04-26T20:32:31.553674+0000","last_undegraded":"2024-04-26T20:32:31.553674+0000","last_fullsized":"2024-04-26T20:32:31.553674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:32.071225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550965+0000","last_change":"2024-04-26T20:32:26.222681+0000","last_active":"2024-04-26T20:32:31.550965+0000","last_peered":"2024-04-26T20:32:31.550965+0000","last_clean":"2024-04-26T20:32:31.550965+0000","last_became_active":"2024-04-26T20:32:26.222099+0000","last_became_peered":"2024-04-26T20:32:26.222099+0000","last_unstale":"2024-04-26T20:32:31.550965+0000","last_undegraded":"2024-04-26T20:32:31.550965+0000","last_fullsized":"2024-04-26T20:32:31.550965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:35:21.958180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:52.669823+0000","last_change":"2024-04-26T20:32:25.276314+0000","last_active":"2024-04-26T20:32:52.669823+0000","last_peered":"2024-04-26T20:32:52.669823+0000","last_clean":"2024-04-26T20:32:52.669823+0000","last_became_active":"2024-04-26T20:32:25.275926+0000","last_became_peered":"2024-04-26T20:32:25.275926+0000","last_unstale":"2024-04-26T20:32:52.669823+0000","last_undegraded":"2024-04-26T20:32:52.669823+0000","last_fullsized":"2024-04-26T20:32:52.669823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:25.027777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361329+0000","last_change":"2024-04-26T20:32:30.385121+0000","last_active":"2024-04-26T20:32:31.361329+0000","last_peered":"2024-04-26T20:32:31.361329+0000","last_clean":"2024-04-26T20:32:31.361329+0000","last_became_active":"2024-04-26T20:32:30.384697+0000","last_became_peered":"2024-04-26T20:32:30.384697+0000","last_unstale":"2024-04-26T20:32:31.361329+0000","last_undegraded":"2024-04-26T20:32:31.361329+0000","last_fullsized":"2024-04-26T20:32:31.361329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:00.725130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.363801+0000","last_change":"2024-04-26T20:32:26.227550+0000","last_active":"2024-04-26T20:32:31.363801+0000","last_peered":"2024-04-26T20:32:31.363801+0000","last_clean":"2024-04-26T20:32:31.363801+0000","last_became_active":"2024-04-26T20:32:26.224262+0000","last_became_peered":"2024-04-26T20:32:26.224262+0000","last_unstale":"2024-04-26T20:32:31.363801+0000","last_undegraded":"2024-04-26T20:32:31.363801+0000","last_fullsized":"2024-04-26T20:32:31.363801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:01.413870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:53.681793+0000","last_change":"2024-04-26T20:32:25.274253+0000","last_active":"2024-04-26T20:32:53.681793+0000","last_peered":"2024-04-26T20:32:53.681793+0000","last_clean":"2024-04-26T20:32:53.681793+0000","last_became_active":"2024-04-26T20:32:25.273956+0000","last_became_peered":"2024-04-26T20:32:25.273956+0000","last_unstale":"2024-04-26T20:32:53.681793+0000","last_undegraded":"2024-04-26T20:32:53.681793+0000","last_fullsized":"2024-04-26T20:32:53.681793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:23:27.832135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":348,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.360757+0000","last_change":"2024-04-26T20:31:05.644347+0000","last_active":"2024-04-26T20:32:31.360757+0000","last_peered":"2024-04-26T20:32:31.360757+0000","last_clean":"2024-04-26T20:32:31.360757+0000","last_became_active":"2024-04-26T20:31:05.643888+0000","last_became_peered":"2024-04-26T20:31:05.643888+0000","last_unstale":"2024-04-26T20:32:31.360757+0000","last_undegraded":"2024-04-26T20:32:31.360757+0000","last_fullsized":"2024-04-26T20:32:31.360757+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:27:11.501328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:27:11.501328+0000","last_clean_scrub_stamp":"2024-04-26T20:27:11.501328+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:08.802687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550797+0000","last_change":"2024-04-26T20:32:30.414972+0000","last_active":"2024-04-26T20:32:31.550797+0000","last_peered":"2024-04-26T20:32:31.550797+0000","last_clean":"2024-04-26T20:32:31.550797+0000","last_became_active":"2024-04-26T20:32:30.414542+0000","last_became_peered":"2024-04-26T20:32:30.414542+0000","last_unstale":"2024-04-26T20:32:31.550797+0000","last_undegraded":"2024-04-26T20:32:31.550797+0000","last_fullsized":"2024-04-26T20:32:31.550797+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:49:06.242986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550900+0000","last_change":"2024-04-26T20:32:26.239936+0000","last_active":"2024-04-26T20:32:31.550900+0000","last_peered":"2024-04-26T20:32:31.550900+0000","last_clean":"2024-04-26T20:32:31.550900+0000","last_became_active":"2024-04-26T20:32:26.239583+0000","last_became_peered":"2024-04-26T20:32:26.239583+0000","last_unstale":"2024-04-26T20:32:31.550900+0000","last_undegraded":"2024-04-26T20:32:31.550900+0000","last_fullsized":"2024-04-26T20:32:31.550900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:57.183460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:52.664139+0000","last_change":"2024-04-26T20:32:25.267876+0000","last_active":"2024-04-26T20:32:52.664139+0000","last_peered":"2024-04-26T20:32:52.664139+0000","last_clean":"2024-04-26T20:32:52.664139+0000","last_became_active":"2024-04-26T20:32:25.267573+0000","last_became_peered":"2024-04-26T20:32:25.267573+0000","last_unstale":"2024-04-26T20:32:52.664139+0000","last_undegraded":"2024-04-26T20:32:52.664139+0000","last_fullsized":"2024-04-26T20:32:52.664139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:43:55.630426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550797+0000","last_change":"2024-04-26T20:32:30.383364+0000","last_active":"2024-04-26T20:32:31.550797+0000","last_peered":"2024-04-26T20:32:31.550797+0000","last_clean":"2024-04-26T20:32:31.550797+0000","last_became_active":"2024-04-26T20:32:30.382875+0000","last_became_peered":"2024-04-26T20:32:30.382875+0000","last_unstale":"2024-04-26T20:32:31.550797+0000","last_undegraded":"2024-04-26T20:32:31.550797+0000","last_fullsized":"2024-04-26T20:32:31.550797+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:55.112888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:53.681035+0000","last_change":"2024-04-26T20:32:25.280508+0000","last_active":"2024-04-26T20:32:53.681035+0000","last_peered":"2024-04-26T20:32:53.681035+0000","last_clean":"2024-04-26T20:32:53.681035+0000","last_became_active":"2024-04-26T20:32:25.280265+0000","last_became_peered":"2024-04-26T20:32:25.280265+0000","last_unstale":"2024-04-26T20:32:53.681035+0000","last_undegraded":"2024-04-26T20:32:53.681035+0000","last_fullsized":"2024-04-26T20:32:53.681035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:16:50.427659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.552664+0000","last_change":"2024-04-26T20:32:26.255230+0000","last_active":"2024-04-26T20:32:31.552664+0000","last_peered":"2024-04-26T20:32:31.552664+0000","last_clean":"2024-04-26T20:32:31.552664+0000","last_became_active":"2024-04-26T20:32:26.254099+0000","last_became_peered":"2024-04-26T20:32:26.254099+0000","last_unstale":"2024-04-26T20:32:31.552664+0000","last_undegraded":"2024-04-26T20:32:31.552664+0000","last_fullsized":"2024-04-26T20:32:31.552664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:41.895816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.363425+0000","last_change":"2024-04-26T20:32:30.414688+0000","last_active":"2024-04-26T20:32:31.363425+0000","last_peered":"2024-04-26T20:32:31.363425+0000","last_clean":"2024-04-26T20:32:31.363425+0000","last_became_active":"2024-04-26T20:32:30.414052+0000","last_became_peered":"2024-04-26T20:32:30.414052+0000","last_unstale":"2024-04-26T20:32:31.363425+0000","last_undegraded":"2024-04-26T20:32:31.363425+0000","last_fullsized":"2024-04-26T20:32:31.363425+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:24.943559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:53.683430+0000","last_change":"2024-04-26T20:32:25.298609+0000","last_active":"2024-04-26T20:32:53.683430+0000","last_peered":"2024-04-26T20:32:53.683430+0000","last_clean":"2024-04-26T20:32:53.683430+0000","last_became_active":"2024-04-26T20:32:25.298155+0000","last_became_peered":"2024-04-26T20:32:25.298155+0000","last_unstale":"2024-04-26T20:32:53.683430+0000","last_undegraded":"2024-04-26T20:32:53.683430+0000","last_fullsized":"2024-04-26T20:32:53.683430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:37.130994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.365759+0000","last_change":"2024-04-26T20:32:26.231203+0000","last_active":"2024-04-26T20:32:31.365759+0000","last_peered":"2024-04-26T20:32:31.365759+0000","last_clean":"2024-04-26T20:32:31.365759+0000","last_became_active":"2024-04-26T20:32:26.230775+0000","last_became_peered":"2024-04-26T20:32:26.230775+0000","last_unstale":"2024-04-26T20:32:31.365759+0000","last_undegraded":"2024-04-26T20:32:31.365759+0000","last_fullsized":"2024-04-26T20:32:31.365759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:40.557744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549252+0000","last_change":"2024-04-26T20:32:30.375703+0000","last_active":"2024-04-26T20:32:31.549252+0000","last_peered":"2024-04-26T20:32:31.549252+0000","last_clean":"2024-04-26T20:32:31.549252+0000","last_became_active":"2024-04-26T20:32:30.375292+0000","last_became_peered":"2024-04-26T20:32:30.375292+0000","last_unstale":"2024-04-26T20:32:31.549252+0000","last_undegraded":"2024-04-26T20:32:31.549252+0000","last_fullsized":"2024-04-26T20:32:31.549252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:39.747426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.658100+0000","last_change":"2024-04-26T20:32:25.290908+0000","last_active":"2024-04-26T20:32:51.658100+0000","last_peered":"2024-04-26T20:32:51.658100+0000","last_clean":"2024-04-26T20:32:51.658100+0000","last_became_active":"2024-04-26T20:32:25.290655+0000","last_became_peered":"2024-04-26T20:32:25.290655+0000","last_unstale":"2024-04-26T20:32:51.658100+0000","last_undegraded":"2024-04-26T20:32:51.658100+0000","last_fullsized":"2024-04-26T20:32:51.658100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:28:42.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.368104+0000","last_change":"2024-04-26T20:32:26.246554+0000","last_active":"2024-04-26T20:32:31.368104+0000","last_peered":"2024-04-26T20:32:31.368104+0000","last_clean":"2024-04-26T20:32:31.368104+0000","last_became_active":"2024-04-26T20:32:26.246279+0000","last_became_peered":"2024-04-26T20:32:26.246279+0000","last_unstale":"2024-04-26T20:32:31.368104+0000","last_undegraded":"2024-04-26T20:32:31.368104+0000","last_fullsized":"2024-04-26T20:32:31.368104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:02.561410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550615+0000","last_change":"2024-04-26T20:32:30.414622+0000","last_active":"2024-04-26T20:32:31.550615+0000","last_peered":"2024-04-26T20:32:31.550615+0000","last_clean":"2024-04-26T20:32:31.550615+0000","last_became_active":"2024-04-26T20:32:30.414176+0000","last_became_peered":"2024-04-26T20:32:30.414176+0000","last_unstale":"2024-04-26T20:32:31.550615+0000","last_undegraded":"2024-04-26T20:32:31.550615+0000","last_fullsized":"2024-04-26T20:32:31.550615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:15.011512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.653091+0000","last_change":"2024-04-26T20:32:25.275220+0000","last_active":"2024-04-26T20:32:51.653091+0000","last_peered":"2024-04-26T20:32:51.653091+0000","last_clean":"2024-04-26T20:32:51.653091+0000","last_became_active":"2024-04-26T20:32:25.274990+0000","last_became_peered":"2024-04-26T20:32:25.274990+0000","last_unstale":"2024-04-26T20:32:51.653091+0000","last_undegraded":"2024-04-26T20:32:51.653091+0000","last_fullsized":"2024-04-26T20:32:51.653091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:58.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548678+0000","last_change":"2024-04-26T20:32:26.218481+0000","last_active":"2024-04-26T20:32:31.548678+0000","last_peered":"2024-04-26T20:32:31.548678+0000","last_clean":"2024-04-26T20:32:31.548678+0000","last_became_active":"2024-04-26T20:32:26.218146+0000","last_became_peered":"2024-04-26T20:32:26.218146+0000","last_unstale":"2024-04-26T20:32:31.548678+0000","last_undegraded":"2024-04-26T20:32:31.548678+0000","last_fullsized":"2024-04-26T20:32:31.548678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:37.310945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551737+0000","last_change":"2024-04-26T20:32:30.383815+0000","last_active":"2024-04-26T20:32:31.551737+0000","last_peered":"2024-04-26T20:32:31.551737+0000","last_clean":"2024-04-26T20:32:31.551737+0000","last_became_active":"2024-04-26T20:32:30.382818+0000","last_became_peered":"2024-04-26T20:32:30.382818+0000","last_unstale":"2024-04-26T20:32:31.551737+0000","last_undegraded":"2024-04-26T20:32:31.551737+0000","last_fullsized":"2024-04-26T20:32:31.551737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:47.839781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.658138+0000","last_change":"2024-04-26T20:32:25.291527+0000","last_active":"2024-04-26T20:32:51.658138+0000","last_peered":"2024-04-26T20:32:51.658138+0000","last_clean":"2024-04-26T20:32:51.658138+0000","last_became_active":"2024-04-26T20:32:25.285353+0000","last_became_peered":"2024-04-26T20:32:25.285353+0000","last_unstale":"2024-04-26T20:32:51.658138+0000","last_undegraded":"2024-04-26T20:32:51.658138+0000","last_fullsized":"2024-04-26T20:32:51.658138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:28.927822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550304+0000","last_change":"2024-04-26T20:32:26.241064+0000","last_active":"2024-04-26T20:32:31.550304+0000","last_peered":"2024-04-26T20:32:31.550304+0000","last_clean":"2024-04-26T20:32:31.550304+0000","last_became_active":"2024-04-26T20:32:26.240673+0000","last_became_peered":"2024-04-26T20:32:26.240673+0000","last_unstale":"2024-04-26T20:32:31.550304+0000","last_undegraded":"2024-04-26T20:32:31.550304+0000","last_fullsized":"2024-04-26T20:32:31.550304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:19:47.777751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550107+0000","last_change":"2024-04-26T20:32:30.388465+0000","last_active":"2024-04-26T20:32:31.550107+0000","last_peered":"2024-04-26T20:32:31.550107+0000","last_clean":"2024-04-26T20:32:31.550107+0000","last_became_active":"2024-04-26T20:32:30.388038+0000","last_became_peered":"2024-04-26T20:32:30.388038+0000","last_unstale":"2024-04-26T20:32:31.550107+0000","last_undegraded":"2024-04-26T20:32:31.550107+0000","last_fullsized":"2024-04-26T20:32:31.550107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:38:39.596970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549755+0000","last_change":"2024-04-26T20:32:25.271914+0000","last_active":"2024-04-26T20:32:31.549755+0000","last_peered":"2024-04-26T20:32:31.549755+0000","last_clean":"2024-04-26T20:32:31.549755+0000","last_became_active":"2024-04-26T20:32:25.271567+0000","last_became_peered":"2024-04-26T20:32:25.271567+0000","last_unstale":"2024-04-26T20:32:31.549755+0000","last_undegraded":"2024-04-26T20:32:31.549755+0000","last_fullsized":"2024-04-26T20:32:31.549755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:02.182470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.364761+0000","last_change":"2024-04-26T20:32:26.245344+0000","last_active":"2024-04-26T20:32:31.364761+0000","last_peered":"2024-04-26T20:32:31.364761+0000","last_clean":"2024-04-26T20:32:31.364761+0000","last_became_active":"2024-04-26T20:32:26.245032+0000","last_became_peered":"2024-04-26T20:32:26.245032+0000","last_unstale":"2024-04-26T20:32:31.364761+0000","last_undegraded":"2024-04-26T20:32:31.364761+0000","last_fullsized":"2024-04-26T20:32:31.364761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:12:51.368935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549938+0000","last_change":"2024-04-26T20:32:30.414408+0000","last_active":"2024-04-26T20:32:31.549938+0000","last_peered":"2024-04-26T20:32:31.549938+0000","last_clean":"2024-04-26T20:32:31.549938+0000","last_became_active":"2024-04-26T20:32:30.413892+0000","last_became_peered":"2024-04-26T20:32:30.413892+0000","last_unstale":"2024-04-26T20:32:31.549938+0000","last_undegraded":"2024-04-26T20:32:31.549938+0000","last_fullsized":"2024-04-26T20:32:31.549938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:53.332840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:51.652652+0000","last_change":"2024-04-26T20:32:25.265440+0000","last_active":"2024-04-26T20:32:51.652652+0000","last_peered":"2024-04-26T20:32:51.652652+0000","last_clean":"2024-04-26T20:32:51.652652+0000","last_became_active":"2024-04-26T20:32:25.265129+0000","last_became_peered":"2024-04-26T20:32:25.265129+0000","last_unstale":"2024-04-26T20:32:51.652652+0000","last_undegraded":"2024-04-26T20:32:51.652652+0000","last_fullsized":"2024-04-26T20:32:51.652652+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:49.475009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361178+0000","last_change":"2024-04-26T20:32:26.234477+0000","last_active":"2024-04-26T20:32:31.361178+0000","last_peered":"2024-04-26T20:32:31.361178+0000","last_clean":"2024-04-26T20:32:31.361178+0000","last_became_active":"2024-04-26T20:32:26.234103+0000","last_became_peered":"2024-04-26T20:32:26.234103+0000","last_unstale":"2024-04-26T20:32:31.361178+0000","last_undegraded":"2024-04-26T20:32:31.361178+0000","last_fullsized":"2024-04-26T20:32:31.361178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:54.098860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548855+0000","last_change":"2024-04-26T20:32:30.391466+0000","last_active":"2024-04-26T20:32:31.548855+0000","last_peered":"2024-04-26T20:32:31.548855+0000","last_clean":"2024-04-26T20:32:31.548855+0000","last_became_active":"2024-04-26T20:32:30.391037+0000","last_became_peered":"2024-04-26T20:32:30.391037+0000","last_unstale":"2024-04-26T20:32:31.548855+0000","last_undegraded":"2024-04-26T20:32:31.548855+0000","last_fullsized":"2024-04-26T20:32:31.548855+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:39:16.305520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550548+0000","last_change":"2024-04-26T20:32:25.281688+0000","last_active":"2024-04-26T20:32:31.550548+0000","last_peered":"2024-04-26T20:32:31.550548+0000","last_clean":"2024-04-26T20:32:31.550548+0000","last_became_active":"2024-04-26T20:32:25.281430+0000","last_became_peered":"2024-04-26T20:32:25.281430+0000","last_unstale":"2024-04-26T20:32:31.550548+0000","last_undegraded":"2024-04-26T20:32:31.550548+0000","last_fullsized":"2024-04-26T20:32:31.550548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:52.785532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361528+0000","last_change":"2024-04-26T20:32:26.226468+0000","last_active":"2024-04-26T20:32:31.361528+0000","last_peered":"2024-04-26T20:32:31.361528+0000","last_clean":"2024-04-26T20:32:31.361528+0000","last_became_active":"2024-04-26T20:32:26.225399+0000","last_became_peered":"2024-04-26T20:32:26.225399+0000","last_unstale":"2024-04-26T20:32:31.361528+0000","last_undegraded":"2024-04-26T20:32:31.361528+0000","last_fullsized":"2024-04-26T20:32:31.361528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:17:11.787658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366950+0000","last_change":"2024-04-26T20:32:30.383646+0000","last_active":"2024-04-26T20:32:31.366950+0000","last_peered":"2024-04-26T20:32:31.366950+0000","last_clean":"2024-04-26T20:32:31.366950+0000","last_became_active":"2024-04-26T20:32:30.381648+0000","last_became_peered":"2024-04-26T20:32:30.381648+0000","last_unstale":"2024-04-26T20:32:31.366950+0000","last_undegraded":"2024-04-26T20:32:31.366950+0000","last_fullsized":"2024-04-26T20:32:31.366950+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:28.036984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551284+0000","last_change":"2024-04-26T20:32:25.275597+0000","last_active":"2024-04-26T20:32:31.551284+0000","last_peered":"2024-04-26T20:32:31.551284+0000","last_clean":"2024-04-26T20:32:31.551284+0000","last_became_active":"2024-04-26T20:32:25.275256+0000","last_became_peered":"2024-04-26T20:32:25.275256+0000","last_unstale":"2024-04-26T20:32:31.551284+0000","last_undegraded":"2024-04-26T20:32:31.551284+0000","last_fullsized":"2024-04-26T20:32:31.551284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:38:20.354879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361187+0000","last_change":"2024-04-26T20:32:26.236885+0000","last_active":"2024-04-26T20:32:31.361187+0000","last_peered":"2024-04-26T20:32:31.361187+0000","last_clean":"2024-04-26T20:32:31.361187+0000","last_became_active":"2024-04-26T20:32:26.236581+0000","last_became_peered":"2024-04-26T20:32:26.236581+0000","last_unstale":"2024-04-26T20:32:31.361187+0000","last_undegraded":"2024-04-26T20:32:31.361187+0000","last_fullsized":"2024-04-26T20:32:31.361187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:17:35.082564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549096+0000","last_change":"2024-04-26T20:32:30.380156+0000","last_active":"2024-04-26T20:32:31.549096+0000","last_peered":"2024-04-26T20:32:31.549096+0000","last_clean":"2024-04-26T20:32:31.549096+0000","last_became_active":"2024-04-26T20:32:30.379240+0000","last_became_peered":"2024-04-26T20:32:30.379240+0000","last_unstale":"2024-04-26T20:32:31.549096+0000","last_undegraded":"2024-04-26T20:32:31.549096+0000","last_fullsized":"2024-04-26T20:32:31.549096+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:42:39.650923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.553339+0000","last_change":"2024-04-26T20:32:25.298850+0000","last_active":"2024-04-26T20:32:31.553339+0000","last_peered":"2024-04-26T20:32:31.553339+0000","last_clean":"2024-04-26T20:32:31.553339+0000","last_became_active":"2024-04-26T20:32:25.298171+0000","last_became_peered":"2024-04-26T20:32:25.298171+0000","last_unstale":"2024-04-26T20:32:31.553339+0000","last_undegraded":"2024-04-26T20:32:31.553339+0000","last_fullsized":"2024-04-26T20:32:31.553339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:17:03.723432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.361349+0000","last_change":"2024-04-26T20:32:26.222873+0000","last_active":"2024-04-26T20:32:31.361349+0000","last_peered":"2024-04-26T20:32:31.361349+0000","last_clean":"2024-04-26T20:32:31.361349+0000","last_became_active":"2024-04-26T20:32:26.222487+0000","last_became_peered":"2024-04-26T20:32:26.222487+0000","last_unstale":"2024-04-26T20:32:31.361349+0000","last_undegraded":"2024-04-26T20:32:31.361349+0000","last_fullsized":"2024-04-26T20:32:31.361349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:17.521537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550408+0000","last_change":"2024-04-26T20:32:26.227092+0000","last_active":"2024-04-26T20:32:31.550408+0000","last_peered":"2024-04-26T20:32:31.550408+0000","last_clean":"2024-04-26T20:32:31.550408+0000","last_became_active":"2024-04-26T20:32:26.226748+0000","last_became_peered":"2024-04-26T20:32:26.226748+0000","last_unstale":"2024-04-26T20:32:31.550408+0000","last_undegraded":"2024-04-26T20:32:31.550408+0000","last_fullsized":"2024-04-26T20:32:31.550408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:25.141131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549875+0000","last_change":"2024-04-26T20:32:26.220388+0000","last_active":"2024-04-26T20:32:31.549875+0000","last_peered":"2024-04-26T20:32:31.549875+0000","last_clean":"2024-04-26T20:32:31.549875+0000","last_became_active":"2024-04-26T20:32:26.219999+0000","last_became_peered":"2024-04-26T20:32:26.219999+0000","last_unstale":"2024-04-26T20:32:31.549875+0000","last_undegraded":"2024-04-26T20:32:31.549875+0000","last_fullsized":"2024-04-26T20:32:31.549875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:11.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.549631+0000","last_change":"2024-04-26T20:32:26.240122+0000","last_active":"2024-04-26T20:32:31.549631+0000","last_peered":"2024-04-26T20:32:31.549631+0000","last_clean":"2024-04-26T20:32:31.549631+0000","last_became_active":"2024-04-26T20:32:26.239746+0000","last_became_peered":"2024-04-26T20:32:26.239746+0000","last_unstale":"2024-04-26T20:32:31.549631+0000","last_undegraded":"2024-04-26T20:32:31.549631+0000","last_fullsized":"2024-04-26T20:32:31.549631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:10.243469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366722+0000","last_change":"2024-04-26T20:32:26.254171+0000","last_active":"2024-04-26T20:32:31.366722+0000","last_peered":"2024-04-26T20:32:31.366722+0000","last_clean":"2024-04-26T20:32:31.366722+0000","last_became_active":"2024-04-26T20:32:26.253775+0000","last_became_peered":"2024-04-26T20:32:26.253775+0000","last_unstale":"2024-04-26T20:32:31.366722+0000","last_undegraded":"2024-04-26T20:32:31.366722+0000","last_fullsized":"2024-04-26T20:32:31.366722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:19:57.968344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.552994+0000","last_change":"2024-04-26T20:32:26.256475+0000","last_active":"2024-04-26T20:32:31.552994+0000","last_peered":"2024-04-26T20:32:31.552994+0000","last_clean":"2024-04-26T20:32:31.552994+0000","last_became_active":"2024-04-26T20:32:26.256119+0000","last_became_peered":"2024-04-26T20:32:26.256119+0000","last_unstale":"2024-04-26T20:32:31.552994+0000","last_undegraded":"2024-04-26T20:32:31.552994+0000","last_fullsized":"2024-04-26T20:32:31.552994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:39:31.456077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550901+0000","last_change":"2024-04-26T20:32:26.229251+0000","last_active":"2024-04-26T20:32:31.550901+0000","last_peered":"2024-04-26T20:32:31.550901+0000","last_clean":"2024-04-26T20:32:31.550901+0000","last_became_active":"2024-04-26T20:32:26.228847+0000","last_became_peered":"2024-04-26T20:32:26.228847+0000","last_unstale":"2024-04-26T20:32:31.550901+0000","last_undegraded":"2024-04-26T20:32:31.550901+0000","last_fullsized":"2024-04-26T20:32:31.550901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:07:28.489681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550455+0000","last_change":"2024-04-26T20:32:26.223939+0000","last_active":"2024-04-26T20:32:31.550455+0000","last_peered":"2024-04-26T20:32:31.550455+0000","last_clean":"2024-04-26T20:32:31.550455+0000","last_became_active":"2024-04-26T20:32:26.223602+0000","last_became_peered":"2024-04-26T20:32:26.223602+0000","last_unstale":"2024-04-26T20:32:31.550455+0000","last_undegraded":"2024-04-26T20:32:31.550455+0000","last_fullsized":"2024-04-26T20:32:31.550455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:54.588411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.366001+0000","last_change":"2024-04-26T20:32:26.229912+0000","last_active":"2024-04-26T20:32:31.366001+0000","last_peered":"2024-04-26T20:32:31.366001+0000","last_clean":"2024-04-26T20:32:31.366001+0000","last_became_active":"2024-04-26T20:32:26.228498+0000","last_became_peered":"2024-04-26T20:32:26.228498+0000","last_unstale":"2024-04-26T20:32:31.366001+0000","last_undegraded":"2024-04-26T20:32:31.366001+0000","last_fullsized":"2024-04-26T20:32:31.366001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:17:53.302225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550578+0000","last_change":"2024-04-26T20:32:26.232087+0000","last_active":"2024-04-26T20:32:31.550578+0000","last_peered":"2024-04-26T20:32:31.550578+0000","last_clean":"2024-04-26T20:32:31.550578+0000","last_became_active":"2024-04-26T20:32:26.231525+0000","last_became_peered":"2024-04-26T20:32:26.231525+0000","last_unstale":"2024-04-26T20:32:31.550578+0000","last_undegraded":"2024-04-26T20:32:31.550578+0000","last_fullsized":"2024-04-26T20:32:31.550578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:05.326866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.364004+0000","last_change":"2024-04-26T20:32:26.224341+0000","last_active":"2024-04-26T20:32:31.364004+0000","last_peered":"2024-04-26T20:32:31.364004+0000","last_clean":"2024-04-26T20:32:31.364004+0000","last_became_active":"2024-04-26T20:32:26.223888+0000","last_became_peered":"2024-04-26T20:32:26.223888+0000","last_unstale":"2024-04-26T20:32:31.364004+0000","last_undegraded":"2024-04-26T20:32:31.364004+0000","last_fullsized":"2024-04-26T20:32:31.364004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:21.103180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.548415+0000","last_change":"2024-04-26T20:32:26.231255+0000","last_active":"2024-04-26T20:32:31.548415+0000","last_peered":"2024-04-26T20:32:31.548415+0000","last_clean":"2024-04-26T20:32:31.548415+0000","last_became_active":"2024-04-26T20:32:26.230823+0000","last_became_peered":"2024-04-26T20:32:26.230823+0000","last_unstale":"2024-04-26T20:32:31.548415+0000","last_undegraded":"2024-04-26T20:32:31.548415+0000","last_fullsized":"2024-04-26T20:32:31.548415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:34.066151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.551796+0000","last_change":"2024-04-26T20:32:26.246452+0000","last_active":"2024-04-26T20:32:31.551796+0000","last_peered":"2024-04-26T20:32:31.551796+0000","last_clean":"2024-04-26T20:32:31.551796+0000","last_became_active":"2024-04-26T20:32:26.246109+0000","last_became_peered":"2024-04-26T20:32:26.246109+0000","last_unstale":"2024-04-26T20:32:31.551796+0000","last_undegraded":"2024-04-26T20:32:31.551796+0000","last_fullsized":"2024-04-26T20:32:31.551796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:42:21.617776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T20:32:31.550782+0000","last_change":"2024-04-26T20:32:26.226274+0000","last_active":"2024-04-26T20:32:31.550782+0000","last_peered":"2024-04-26T20:32:31.550782+0000","last_clean":"2024-04-26T20:32:31.550782+0000","last_became_active":"2024-04-26T20:32:26.225751+0000","last_became_peered":"2024-04-26T20:32:26.225751+0000","last_unstale":"2024-04-26T20:32:31.550782+0000","last_undegraded":"2024-04-26T20:32:31.550782+0000","last_fullsized":"2024-04-26T20:32:31.550782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:39.050161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8326,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":22890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1072381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":482433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070501,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":477953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332137,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1068787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":481091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":477881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1605632,"data_stored":1069059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T20:32:57.943 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T20:32:57.943 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T20:32:57.944 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T20:32:58.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:57 smithi112 ceph-mon[32769]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:57 smithi112 ceph-mon[32769]: pgmap v333: 129 pgs: 129 active+clean; 585 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 9 op/s 2024-04-26T20:32:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:57 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2945108522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:58.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:57 smithi130 ceph-mon[32704]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:32:58.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:57 smithi130 ceph-mon[32704]: pgmap v333: 129 pgs: 129 active+clean; 585 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 9 op/s 2024-04-26T20:32:58.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:57 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2945108522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:32:58.428 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:32:58.836 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:58 smithi102 ceph-mon[28094]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:32:59.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:58 smithi112 ceph-mon[32769]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:32:59.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:58 smithi130 ceph-mon[32704]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:32:59.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:59 smithi102 ceph-mon[28094]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:32:59.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:32:59 smithi102 ceph-mon[28094]: pgmap v334: 129 pgs: 129 active+clean; 622 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 7.8 KiB/s wr, 16 op/s 2024-04-26T20:33:00.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:59 smithi112 ceph-mon[32769]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:33:00.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:32:59 smithi112 ceph-mon[32769]: pgmap v334: 129 pgs: 129 active+clean; 622 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 7.8 KiB/s wr, 16 op/s 2024-04-26T20:33:00.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:59 smithi130 ceph-mon[32704]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:33:00.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:32:59 smithi130 ceph-mon[32704]: pgmap v334: 129 pgs: 129 active+clean; 622 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 7.8 KiB/s wr, 16 op/s 2024-04-26T20:33:00.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.959+0000 7f4ecec00700 1 -- 172.21.15.102:0/535445300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec8075e20 msgr2=0x7f4ec80762a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:00.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.959+0000 7f4ecec00700 1 --2- 172.21.15.102:0/535445300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec8075e20 0x7f4ec80762a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40098f0 tx=0x7f4eb4009c00 comp rx=0 tx=0).stop 2024-04-26T20:33:00.961 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:00 smithi102 ceph-mon[28094]: pgmap v335: 129 pgs: 129 active+clean; 623 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.9 KiB/s wr, 20 op/s 2024-04-26T20:33:00.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.959+0000 7f4ecec00700 1 -- 172.21.15.102:0/535445300 shutdown_connections 2024-04-26T20:33:00.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.959+0000 7f4ecec00700 1 --2- 172.21.15.102:0/535445300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ec810b1c0 0x7f4ec810d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:00.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.959+0000 7f4ecec00700 1 --2- 172.21.15.102:0/535445300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ec80773f0 0x7f4ec8075850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:00.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.959+0000 7f4ecec00700 1 --2- 172.21.15.102:0/535445300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec8075e20 0x7f4ec80762a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:00.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.959+0000 7f4ecec00700 1 -- 172.21.15.102:0/535445300 >> 172.21.15.102:0/535445300 conn(0x7f4ec80fe860 msgr2=0x7f4ec8100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:00.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.959+0000 7f4ecec00700 1 -- 172.21.15.102:0/535445300 shutdown_connections 2024-04-26T20:33:00.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.959+0000 7f4ecec00700 1 -- 172.21.15.102:0/535445300 wait complete. 2024-04-26T20:33:00.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.961+0000 7f4ecec00700 1 Processor -- start 2024-04-26T20:33:00.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.961+0000 7f4ecec00700 1 -- start start 2024-04-26T20:33:00.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.961+0000 7f4ecec00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec8075e20 0x7f4ec819b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:00.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.961+0000 7f4ecec00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ec80773f0 0x7f4ec819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:00.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.961+0000 7f4ecec00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ec810b1c0 0x7f4ec81a2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:00.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.961+0000 7f4ecec00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec8110f90 con 0x7f4ec8075e20 2024-04-26T20:33:00.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.961+0000 7f4ecec00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec8110e10 con 0x7f4ec80773f0 2024-04-26T20:33:00.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.961+0000 7f4ecec00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec8111110 con 0x7f4ec810b1c0 2024-04-26T20:33:00.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.962+0000 7f4ecd800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec8075e20 0x7f4ec819b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:00.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.962+0000 7f4ecd800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec8075e20 0x7f4ec819b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33186/0 (socket says 172.21.15.102:33186) 2024-04-26T20:33:00.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.962+0000 7f4ecd800700 1 -- 172.21.15.102:0/1816431254 learned_addr learned my addr 172.21.15.102:0/1816431254 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.962+0000 7f4ece200700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ec810b1c0 0x7f4ec81a2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.962+0000 7f4ecce00700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ec80773f0 0x7f4ec819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.962+0000 7f4ecce00700 1 -- 172.21.15.102:0/1816431254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ec810b1c0 msgr2=0x7f4ec81a2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.962+0000 7f4ecce00700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ec810b1c0 0x7f4ec81a2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.962+0000 7f4ecce00700 1 -- 172.21.15.102:0/1816431254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec8075e20 msgr2=0x7f4ec819b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.962+0000 7f4ecce00700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec8075e20 0x7f4ec819b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.962+0000 7f4ecce00700 1 -- 172.21.15.102:0/1816431254 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec81a31f0 con 0x7f4ec80773f0 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.963+0000 7f4ecce00700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ec80773f0 0x7f4ec819bad0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009230 tx=0x7f4eb4005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.963+0000 7f4ec6000700 1 -- 172.21.15.102:0/1816431254 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4005510 con 0x7f4ec80773f0 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.963+0000 7f4ec6000700 1 -- 172.21.15.102:0/1816431254 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eb40056b0 con 0x7f4ec80773f0 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.963+0000 7f4ecec00700 1 -- 172.21.15.102:0/1816431254 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec81a3480 con 0x7f4ec80773f0 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.963+0000 7f4ec6000700 1 -- 172.21.15.102:0/1816431254 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4003a60 con 0x7f4ec80773f0 2024-04-26T20:33:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.963+0000 7f4ecec00700 1 -- 172.21.15.102:0/1816431254 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec81a38d0 con 0x7f4ec80773f0 2024-04-26T20:33:00.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.964+0000 7f4ecec00700 1 -- 172.21.15.102:0/1816431254 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec8064a80 con 0x7f4ec80773f0 2024-04-26T20:33:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.966+0000 7f4ec6000700 1 -- 172.21.15.102:0/1816431254 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f4eb4003c40 con 0x7f4ec80773f0 2024-04-26T20:33:00.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.967+0000 7f4ec6000700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4eb00a3d80 0x7f4eb00a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.968+0000 7f4ec6000700 1 -- 172.21.15.102:0/1816431254 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4eb40d1740 con 0x7f4ec80773f0 2024-04-26T20:33:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.968+0000 7f4ecd800700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4eb00a3d80 0x7f4eb00a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.968+0000 7f4ecd800700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4eb00a3d80 0x7f4eb00a6240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc005950 tx=0x7f4ebc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:00.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:00.972+0000 7f4ec6000700 1 -- 172.21.15.102:0/1816431254 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4eb4013030 con 0x7f4ec80773f0 2024-04-26T20:33:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:00 smithi112 ceph-mon[32769]: pgmap v335: 129 pgs: 129 active+clean; 623 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.9 KiB/s wr, 20 op/s 2024-04-26T20:33:01.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:00 smithi130 ceph-mon[32704]: pgmap v335: 129 pgs: 129 active+clean; 623 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.9 KiB/s wr, 20 op/s 2024-04-26T20:33:01.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.284+0000 7f4ecec00700 1 -- 172.21.15.102:0/1816431254 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f4ec802e640 con 0x7f4ec80773f0 2024-04-26T20:33:01.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.293+0000 7f4ec6000700 1 -- 172.21.15.102:0/1816431254 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f4ec802e640 con 0x7f4ec80773f0 2024-04-26T20:33:01.295 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-04-26T20:33:01.295 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQB9DyxmAmEmERAAmdkNG0bKaNU1iyow2xdhsQ== 2024-04-26T20:33:01.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.296+0000 7f4ecec00700 1 -- 172.21.15.102:0/1816431254 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4eb00a3d80 msgr2=0x7f4eb00a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:01.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.296+0000 7f4ecec00700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4eb00a3d80 0x7f4eb00a6240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc005950 tx=0x7f4ebc009310 comp rx=0 tx=0).stop 2024-04-26T20:33:01.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.296+0000 7f4ecec00700 1 -- 172.21.15.102:0/1816431254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ec80773f0 msgr2=0x7f4ec819bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:01.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.296+0000 7f4ecec00700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ec80773f0 0x7f4ec819bad0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009230 tx=0x7f4eb4005270 comp rx=0 tx=0).stop 2024-04-26T20:33:01.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.297+0000 7f4ecec00700 1 -- 172.21.15.102:0/1816431254 shutdown_connections 2024-04-26T20:33:01.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.297+0000 7f4ecec00700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f4eb00a3d80 0x7f4eb00a6240 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:01.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.297+0000 7f4ecec00700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ec810b1c0 0x7f4ec81a2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:01.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.297+0000 7f4ecec00700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ec8075e20 0x7f4ec819b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.297+0000 7f4ecec00700 1 --2- 172.21.15.102:0/1816431254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ec80773f0 0x7f4ec819bad0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.297+0000 7f4ecec00700 1 -- 172.21.15.102:0/1816431254 >> 172.21.15.102:0/1816431254 conn(0x7f4ec80fe860 msgr2=0x7f4ec8107110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.297+0000 7f4ecec00700 1 -- 172.21.15.102:0/1816431254 shutdown_connections 2024-04-26T20:33:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:01.297+0000 7f4ecec00700 1 -- 172.21.15.102:0/1816431254 wait complete. 2024-04-26T20:33:02.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:01 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1816431254' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:33:02.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:01 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:33:02.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:01 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T20:33:02.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:01 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1816431254' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:33:02.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:01 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:33:02.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:01 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T20:33:02.099 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:33:02.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T20:33:02.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T20:33:02.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:01 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1816431254' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:33:02.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:01 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T20:33:02.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:01 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T20:33:02.235 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T20:33:02.235 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T20:33:02.235 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T20:33:02.735 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:03.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:02 smithi112 ceph-mon[32769]: pgmap v336: 129 pgs: 129 active+clean; 623 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.9 KiB/s wr, 20 op/s 2024-04-26T20:33:03.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:02 smithi102 ceph-mon[28094]: pgmap v336: 129 pgs: 129 active+clean; 623 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.9 KiB/s wr, 20 op/s 2024-04-26T20:33:03.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:02 smithi130 ceph-mon[32704]: pgmap v336: 129 pgs: 129 active+clean; 623 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.9 KiB/s wr, 20 op/s 2024-04-26T20:33:04.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.969+0000 7f0a75a00700 1 -- 172.21.15.102:0/1790197902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a701045e0 msgr2=0x7f0a701049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:04.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.969+0000 7f0a75a00700 1 --2- 172.21.15.102:0/1790197902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a701045e0 0x7f0a701049c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009990 tx=0x7f0a60009ca0 comp rx=0 tx=0).stop 2024-04-26T20:33:04.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.969+0000 7f0a75a00700 1 -- 172.21.15.102:0/1790197902 shutdown_connections 2024-04-26T20:33:04.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.969+0000 7f0a75a00700 1 --2- 172.21.15.102:0/1790197902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a7010d9c0 0x7f0a7010fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:04.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.969+0000 7f0a75a00700 1 --2- 172.21.15.102:0/1790197902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a701045e0 0x7f0a701049c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:04.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.969+0000 7f0a75a00700 1 --2- 172.21.15.102:0/1790197902 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a70104f00 0x7f0a7010d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:04.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.969+0000 7f0a75a00700 1 -- 172.21.15.102:0/1790197902 >> 172.21.15.102:0/1790197902 conn(0x7f0a700fddb0 msgr2=0x7f0a701001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:04.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.969+0000 7f0a75a00700 1 -- 172.21.15.102:0/1790197902 shutdown_connections 2024-04-26T20:33:04.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.970+0000 7f0a75a00700 1 -- 172.21.15.102:0/1790197902 wait complete. 2024-04-26T20:33:04.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.971+0000 7f0a75a00700 1 Processor -- start 2024-04-26T20:33:04.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.971+0000 7f0a75a00700 1 -- start start 2024-04-26T20:33:04.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a75a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a701045e0 0x7f0a70102240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a75a00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a70104f00 0x7f0a70102780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a75a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7010d9c0 0x7f0a70102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a75a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0a701135b0 con 0x7f0a701045e0 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a75a00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0a70113430 con 0x7f0a70104f00 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a75a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0a70113730 con 0x7f0a7010d9c0 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a6fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a701045e0 0x7f0a70102240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a6fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a701045e0 0x7f0a70102240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50408/0 (socket says 172.21.15.102:50408) 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a6fe00700 1 -- 172.21.15.102:0/4246711828 learned_addr learned my addr 172.21.15.102:0/4246711828 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a67e00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a70104f00 0x7f0a70102780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.972+0000 7f0a75000700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7010d9c0 0x7f0a70102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:04.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.973+0000 7f0a6fe00700 1 -- 172.21.15.102:0/4246711828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7010d9c0 msgr2=0x7f0a70102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:04.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.973+0000 7f0a6fe00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7010d9c0 0x7f0a70102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:04.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.973+0000 7f0a6fe00700 1 -- 172.21.15.102:0/4246711828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a70104f00 msgr2=0x7f0a70102780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:04.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.973+0000 7f0a6fe00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a70104f00 0x7f0a70102780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:04.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.973+0000 7f0a6fe00700 1 -- 172.21.15.102:0/4246711828 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a701a8080 con 0x7f0a701045e0 2024-04-26T20:33:04.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.973+0000 7f0a67e00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a70104f00 0x7f0a70102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:04.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.973+0000 7f0a6fe00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a701045e0 0x7f0a70102240 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0a60000c00 tx=0x7f0a6000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:04.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.973+0000 7f0a6d600700 1 -- 172.21.15.102:0/4246711828 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60027070 con 0x7f0a701045e0 2024-04-26T20:33:04.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.973+0000 7f0a75a00700 1 -- 172.21.15.102:0/4246711828 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a701a8310 con 0x7f0a701045e0 2024-04-26T20:33:04.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.973+0000 7f0a6d600700 1 -- 172.21.15.102:0/4246711828 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a60022470 con 0x7f0a701045e0 2024-04-26T20:33:04.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.974+0000 7f0a6d600700 1 -- 172.21.15.102:0/4246711828 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60017650 con 0x7f0a701045e0 2024-04-26T20:33:04.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.974+0000 7f0a75a00700 1 -- 172.21.15.102:0/4246711828 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a701a87f0 con 0x7f0a701045e0 2024-04-26T20:33:04.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.975+0000 7f0a75a00700 1 -- 172.21.15.102:0/4246711828 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a7006a110 con 0x7f0a701045e0 2024-04-26T20:33:04.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.977+0000 7f0a6d600700 1 -- 172.21.15.102:0/4246711828 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f0a6002a070 con 0x7f0a701045e0 2024-04-26T20:33:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.978+0000 7f0a6d600700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0a540a3cd0 0x7f0a540a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.979+0000 7f0a6d600700 1 -- 172.21.15.102:0/4246711828 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0a600d1590 con 0x7f0a701045e0 2024-04-26T20:33:04.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.979+0000 7f0a67e00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0a540a3cd0 0x7f0a540a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:04.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.979+0000 7f0a67e00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0a540a3cd0 0x7f0a540a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0a7010ee00 tx=0x7f0a58009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:04.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:04.984+0000 7f0a6d600700 1 -- 172.21.15.102:0/4246711828 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0a60013070 con 0x7f0a701045e0 2024-04-26T20:33:05.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.270+0000 7f0a75a00700 1 -- 172.21.15.102:0/4246711828 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f0a7002e490 con 0x7f0a701045e0 2024-04-26T20:33:05.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.288+0000 7f0a6d600700 1 -- 172.21.15.102:0/4246711828 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+265000 (secure 0 0 0) 0x7f0a6009dc00 con 0x7f0a701045e0 2024-04-26T20:33:05.296 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:33:05.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 -- 172.21.15.102:0/4246711828 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0a540a3cd0 msgr2=0x7f0a540a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:05.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0a540a3cd0 0x7f0a540a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0a7010ee00 tx=0x7f0a58009260 comp rx=0 tx=0).stop 2024-04-26T20:33:05.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 -- 172.21.15.102:0/4246711828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a701045e0 msgr2=0x7f0a70102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:05.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a701045e0 0x7f0a70102240 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0a60000c00 tx=0x7f0a6000bfd0 comp rx=0 tx=0).stop 2024-04-26T20:33:05.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 -- 172.21.15.102:0/4246711828 shutdown_connections 2024-04-26T20:33:05.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0a540a3cd0 0x7f0a540a6190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:05.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7010d9c0 0x7f0a70102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a701045e0 0x7f0a70102240 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 --2- 172.21.15.102:0/4246711828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a70104f00 0x7f0a70102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 -- 172.21.15.102:0/4246711828 >> 172.21.15.102:0/4246711828 conn(0x7f0a700fddb0 msgr2=0x7f0a700fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:05.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 -- 172.21.15.102:0/4246711828 shutdown_connections 2024-04-26T20:33:05.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:05.293+0000 7f0a75a00700 1 -- 172.21.15.102:0/4246711828 wait complete. 2024-04-26T20:33:05.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:05 smithi130 ceph-mon[32704]: pgmap v337: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.0 KiB/s wr, 24 op/s 2024-04-26T20:33:05.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:05 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/4246711828' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T20:33:05.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:05 smithi102 ceph-mon[28094]: pgmap v337: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.0 KiB/s wr, 24 op/s 2024-04-26T20:33:05.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:05 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/4246711828' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T20:33:06.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:05 smithi112 ceph-mon[32769]: pgmap v337: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.0 KiB/s wr, 24 op/s 2024-04-26T20:33:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:05 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/4246711828' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T20:33:06.159 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":71145250},{"type":"v1","addr":"172.21.15.102:6801","nonce":71145250}]},"active_addr":"172.21.15.102:6801/71145250","active_change":"2024-04-26T20:24:28.376556+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.102:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3337636378}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3464299584}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3448933696}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":788432019}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2537603334}]}]} 2024-04-26T20:33:06.165 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T20:33:06.165 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T20:33:06.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T20:33:06.619 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:07.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:07 smithi130 ceph-mon[32704]: pgmap v338: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-04-26T20:33:07.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:07 smithi102 ceph-mon[28094]: pgmap v338: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-04-26T20:33:08.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:07 smithi112 ceph-mon[32769]: pgmap v338: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-04-26T20:33:08.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.790+0000 7f7725800700 1 -- 172.21.15.102:0/3141303829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7720109a50 msgr2=0x7f7720109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:08.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.790+0000 7f7725800700 1 --2- 172.21.15.102:0/3141303829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7720109a50 0x7f7720109ed0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7708009990 tx=0x7f7708009ca0 comp rx=0 tx=0).stop 2024-04-26T20:33:08.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.790+0000 7f7725800700 1 -- 172.21.15.102:0/3141303829 shutdown_connections 2024-04-26T20:33:08.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.790+0000 7f7725800700 1 --2- 172.21.15.102:0/3141303829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010a410 0x7f7720116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:08.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.790+0000 7f7725800700 1 --2- 172.21.15.102:0/3141303829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f772010fa50 0x7f772010fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:08.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.790+0000 7f7725800700 1 --2- 172.21.15.102:0/3141303829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7720109a50 0x7f7720109ed0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:08.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.790+0000 7f7725800700 1 -- 172.21.15.102:0/3141303829 >> 172.21.15.102:0/3141303829 conn(0x7f7720076bf0 msgr2=0x7f7720108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:08.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.790+0000 7f7725800700 1 -- 172.21.15.102:0/3141303829 shutdown_connections 2024-04-26T20:33:08.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.790+0000 7f7725800700 1 -- 172.21.15.102:0/3141303829 wait complete. 2024-04-26T20:33:08.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.791+0000 7f7725800700 1 Processor -- start 2024-04-26T20:33:08.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.791+0000 7f7725800700 1 -- start start 2024-04-26T20:33:08.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f7725800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7720109a50 0x7f77201a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:08.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f7725800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f772010a410 0x7f77201a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:08.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f7725800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010fa50 0x7f77201a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:08.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f7725800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7720119ca0 con 0x7f7720109a50 2024-04-26T20:33:08.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f7725800700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7720119b20 con 0x7f772010fa50 2024-04-26T20:33:08.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f7725800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7720119e20 con 0x7f772010a410 2024-04-26T20:33:08.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f771ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7720109a50 0x7f77201a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:08.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f771ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7720109a50 0x7f77201a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50414/0 (socket says 172.21.15.102:50414) 2024-04-26T20:33:08.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f771ea00700 1 -- 172.21.15.102:0/3295956425 learned_addr learned my addr 172.21.15.102:0/3295956425 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:08.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f7717e00700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f772010a410 0x7f77201a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:08.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.792+0000 7f771f400700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010fa50 0x7f77201a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.793+0000 7f771ea00700 1 -- 172.21.15.102:0/3295956425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f772010a410 msgr2=0x7f77201a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.793+0000 7f771ea00700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f772010a410 0x7f77201a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.793+0000 7f771ea00700 1 -- 172.21.15.102:0/3295956425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010fa50 msgr2=0x7f77201a4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.793+0000 7f771ea00700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010fa50 0x7f77201a4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.793+0000 7f771ea00700 1 -- 172.21.15.102:0/3295956425 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77201a5140 con 0x7f7720109a50 2024-04-26T20:33:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.793+0000 7f771ea00700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7720109a50 0x7f77201a0130 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f771000e9c0 tx=0x7f771000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.793+0000 7f7717400700 1 -- 172.21.15.102:0/3295956425 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771000cbd0 con 0x7f7720109a50 2024-04-26T20:33:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.793+0000 7f7717400700 1 -- 172.21.15.102:0/3295956425 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f771000cd70 con 0x7f7720109a50 2024-04-26T20:33:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.793+0000 7f7717400700 1 -- 172.21.15.102:0/3295956425 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710005610 con 0x7f7720109a50 2024-04-26T20:33:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.793+0000 7f7725800700 1 -- 172.21.15.102:0/3295956425 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77201a5430 con 0x7f7720109a50 2024-04-26T20:33:08.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.794+0000 7f7725800700 1 -- 172.21.15.102:0/3295956425 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77201accb0 con 0x7f7720109a50 2024-04-26T20:33:08.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.795+0000 7f7725800700 1 -- 172.21.15.102:0/3295956425 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f772006a110 con 0x7f7720109a50 2024-04-26T20:33:08.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.797+0000 7f7717400700 1 -- 172.21.15.102:0/3295956425 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f7710004750 con 0x7f7720109a50 2024-04-26T20:33:08.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.799+0000 7f7717400700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77040a3cd0 0x7f77040a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:08.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.799+0000 7f7717400700 1 -- 172.21.15.102:0/3295956425 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f77100d0ab0 con 0x7f7720109a50 2024-04-26T20:33:08.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.799+0000 7f7717e00700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77040a3cd0 0x7f77040a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:08.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.800+0000 7f7717e00700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77040a3cd0 0x7f77040a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f770800b2e0 tx=0x7f770801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:08.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:08.804+0000 7f7717400700 1 -- 172.21.15.102:0/3295956425 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f771009d090 con 0x7f7720109a50 2024-04-26T20:33:09.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.033+0000 7f7725800700 1 -- 172.21.15.102:0/3295956425 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77201a14c0 con 0x7f7720109a50 2024-04-26T20:33:09.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.038+0000 7f7717400700 1 -- 172.21.15.102:0/3295956425 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24164 (secure 0 0 0) 0x7f77201a14c0 con 0x7f7720109a50 2024-04-26T20:33:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:33:09.039 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:32:30.827511+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:33:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.040+0000 7f7725800700 1 -- 172.21.15.102:0/3295956425 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77040a3cd0 msgr2=0x7f77040a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.040+0000 7f7725800700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77040a3cd0 0x7f77040a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f770800b2e0 tx=0x7f770801a040 comp rx=0 tx=0).stop 2024-04-26T20:33:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.040+0000 7f7725800700 1 -- 172.21.15.102:0/3295956425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7720109a50 msgr2=0x7f77201a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.040+0000 7f7725800700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7720109a50 0x7f77201a0130 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f771000e9c0 tx=0x7f771000ecd0 comp rx=0 tx=0).stop 2024-04-26T20:33:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.040+0000 7f7725800700 1 -- 172.21.15.102:0/3295956425 shutdown_connections 2024-04-26T20:33:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.040+0000 7f7725800700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f77040a3cd0 0x7f77040a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.040+0000 7f7725800700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010fa50 0x7f77201a4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.040+0000 7f7725800700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7720109a50 0x7f77201a0130 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:09.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.040+0000 7f7725800700 1 --2- 172.21.15.102:0/3295956425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f772010a410 0x7f77201a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.040+0000 7f7725800700 1 -- 172.21.15.102:0/3295956425 >> 172.21.15.102:0/3295956425 conn(0x7f7720076bf0 msgr2=0x7f77200771b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.041+0000 7f7725800700 1 -- 172.21.15.102:0/3295956425 shutdown_connections 2024-04-26T20:33:09.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:09.041+0000 7f7725800700 1 -- 172.21.15.102:0/3295956425 wait complete. 2024-04-26T20:33:09.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:09 smithi102 ceph-mon[28094]: pgmap v339: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 KiB/s wr, 19 op/s 2024-04-26T20:33:09.796 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:09 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3295956425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:33:09.862 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T20:33:09.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T20:33:09.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:09 smithi130 ceph-mon[32704]: pgmap v339: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 KiB/s wr, 19 op/s 2024-04-26T20:33:09.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:09 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3295956425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:33:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:09 smithi112 ceph-mon[32769]: pgmap v339: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 KiB/s wr, 19 op/s 2024-04-26T20:33:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:09 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3295956425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:33:10.312 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:11.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:11 smithi102 ceph-mon[28094]: pgmap v340: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-26T20:33:11.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:11 smithi130 ceph-mon[32704]: pgmap v340: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-26T20:33:12.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:11 smithi112 ceph-mon[32769]: pgmap v340: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-26T20:33:12.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.626+0000 7fb5c1a00700 1 -- 172.21.15.102:0/2686101556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5bc10fa50 msgr2=0x7fb5bc10fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:12.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.626+0000 7fb5c1a00700 1 --2- 172.21.15.102:0/2686101556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5bc10fa50 0x7fb5bc10fe30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00b0a0 tx=0x7fb5ac00b3b0 comp rx=0 tx=0).stop 2024-04-26T20:33:12.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.626+0000 7fb5c1a00700 1 -- 172.21.15.102:0/2686101556 shutdown_connections 2024-04-26T20:33:12.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.626+0000 7fb5c1a00700 1 --2- 172.21.15.102:0/2686101556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb5bc10a410 0x7fb5bc116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:12.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.626+0000 7fb5c1a00700 1 --2- 172.21.15.102:0/2686101556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5bc10fa50 0x7fb5bc10fe30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:12.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.626+0000 7fb5c1a00700 1 --2- 172.21.15.102:0/2686101556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5bc109a50 0x7fb5bc109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:12.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.626+0000 7fb5c1a00700 1 -- 172.21.15.102:0/2686101556 >> 172.21.15.102:0/2686101556 conn(0x7fb5bc076bf0 msgr2=0x7fb5bc108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:12.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.626+0000 7fb5c1a00700 1 -- 172.21.15.102:0/2686101556 shutdown_connections 2024-04-26T20:33:12.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.627+0000 7fb5c1a00700 1 -- 172.21.15.102:0/2686101556 wait complete. 2024-04-26T20:33:12.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.628+0000 7fb5c1a00700 1 Processor -- start 2024-04-26T20:33:12.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.628+0000 7fb5c1a00700 1 -- start start 2024-04-26T20:33:12.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.628+0000 7fb5c1a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5bc109a50 0x7fb5bc1a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:12.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.628+0000 7fb5c1a00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb5bc10a410 0x7fb5bc1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.628+0000 7fb5c1a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5bc10fa50 0x7fb5bc1a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.628+0000 7fb5c1a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc119c50 con 0x7fb5bc109a50 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.628+0000 7fb5c1a00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc119ad0 con 0x7fb5bc10a410 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.628+0000 7fb5c1a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc119dd0 con 0x7fb5bc10fa50 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.629+0000 7fb5baa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5bc109a50 0x7fb5bc1a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.629+0000 7fb5b2000700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb5bc10a410 0x7fb5bc1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.629+0000 7fb5baa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5bc109a50 0x7fb5bc1a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50432/0 (socket says 172.21.15.102:50432) 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.629+0000 7fb5baa00700 1 -- 172.21.15.102:0/196463792 learned_addr learned my addr 172.21.15.102:0/196463792 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.629+0000 7fb5bb400700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5bc10fa50 0x7fb5bc1a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.629+0000 7fb5b2000700 1 -- 172.21.15.102:0/196463792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5bc10fa50 msgr2=0x7fb5bc1a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.629+0000 7fb5b2000700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5bc10fa50 0x7fb5bc1a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.629+0000 7fb5b2000700 1 -- 172.21.15.102:0/196463792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5bc109a50 msgr2=0x7fb5bc1a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.629+0000 7fb5b2000700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5bc109a50 0x7fb5bc1a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.629+0000 7fb5b2000700 1 -- 172.21.15.102:0/196463792 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5bc1a5120 con 0x7fb5bc10a410 2024-04-26T20:33:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.630+0000 7fb5b2000700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb5bc10a410 0x7fb5bc1a06a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400e9f0 tx=0x7fb5a400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.630+0000 7fb5b3e00700 1 -- 172.21.15.102:0/196463792 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4004220 con 0x7fb5bc10a410 2024-04-26T20:33:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.630+0000 7fb5b3e00700 1 -- 172.21.15.102:0/196463792 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a40043c0 con 0x7fb5bc10a410 2024-04-26T20:33:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.630+0000 7fb5c1a00700 1 -- 172.21.15.102:0/196463792 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5bc1a5410 con 0x7fb5bc10a410 2024-04-26T20:33:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.630+0000 7fb5b3e00700 1 -- 172.21.15.102:0/196463792 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4005020 con 0x7fb5bc10a410 2024-04-26T20:33:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.630+0000 7fb5c1a00700 1 -- 172.21.15.102:0/196463792 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5bc1acc90 con 0x7fb5bc10a410 2024-04-26T20:33:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.631+0000 7fb5c1a00700 1 -- 172.21.15.102:0/196463792 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5bc06a110 con 0x7fb5bc10a410 2024-04-26T20:33:12.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.633+0000 7fb5b3e00700 1 -- 172.21.15.102:0/196463792 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fb5a4004750 con 0x7fb5bc10a410 2024-04-26T20:33:12.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.634+0000 7fb5b3e00700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5a00a3d50 0x7fb5a00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:12.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.635+0000 7fb5b3e00700 1 -- 172.21.15.102:0/196463792 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb5a40d06e0 con 0x7fb5bc10a410 2024-04-26T20:33:12.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.635+0000 7fb5baa00700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5a00a3d50 0x7fb5a00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:12.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.635+0000 7fb5baa00700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5a00a3d50 0x7fb5a00a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009e60 tx=0x7fb5ac00b740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.638+0000 7fb5b3e00700 1 -- 172.21.15.102:0/196463792 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb5a409ccc0 con 0x7fb5bc10a410 2024-04-26T20:33:12.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.871+0000 7fb5c1a00700 1 -- 172.21.15.102:0/196463792 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5bc1a1470 con 0x7fb5bc10a410 2024-04-26T20:33:12.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.874+0000 7fb5b3e00700 1 -- 172.21.15.102:0/196463792 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24164 (secure 0 0 0) 0x7fb5a409c5c0 con 0x7fb5bc10a410 2024-04-26T20:33:12.875 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:33:12.876 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:32:30.827511+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:33:12.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 -- 172.21.15.102:0/196463792 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5a00a3d50 msgr2=0x7fb5a00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:12.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5a00a3d50 0x7fb5a00a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009e60 tx=0x7fb5ac00b740 comp rx=0 tx=0).stop 2024-04-26T20:33:12.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 -- 172.21.15.102:0/196463792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb5bc10a410 msgr2=0x7fb5bc1a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:12.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb5bc10a410 0x7fb5bc1a06a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400e9f0 tx=0x7fb5a400ed00 comp rx=0 tx=0).stop 2024-04-26T20:33:12.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 -- 172.21.15.102:0/196463792 shutdown_connections 2024-04-26T20:33:12.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5a00a3d50 0x7fb5a00a6210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:12.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5bc10fa50 0x7fb5bc1a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:12.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5bc109a50 0x7fb5bc1a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:12.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 --2- 172.21.15.102:0/196463792 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb5bc10a410 0x7fb5bc1a06a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:12.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 -- 172.21.15.102:0/196463792 >> 172.21.15.102:0/196463792 conn(0x7fb5bc076bf0 msgr2=0x7fb5bc0771f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:12.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 -- 172.21.15.102:0/196463792 shutdown_connections 2024-04-26T20:33:12.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:12.878+0000 7fb5c1a00700 1 -- 172.21.15.102:0/196463792 wait complete. 2024-04-26T20:33:13.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T20:33:13.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T20:33:13.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T20:33:13.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T20:33:13.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T20:33:13.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T20:33:13.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T20:33:13.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T20:33:13.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T20:33:13.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T20:33:13.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T20:33:13.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T20:33:13.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:13 smithi102 ceph-mon[28094]: pgmap v341: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T20:33:13.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:13 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/196463792' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:33:13.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:13 smithi130 ceph-mon[32704]: pgmap v341: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T20:33:13.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:13 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/196463792' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:33:14.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:13 smithi112 ceph-mon[32769]: pgmap v341: 129 pgs: 129 active+clean; 631 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-26T20:33:14.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:13 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/196463792' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:33:14.694 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:14.733 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:14.757 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:14.795 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:14.808 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:14.817 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:15.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:15 smithi130 ceph-mon[32704]: pgmap v342: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-26T20:33:15.892 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:15.898 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:15.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:15 smithi102 ceph-mon[28094]: pgmap v342: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-26T20:33:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:16.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:15 smithi112 ceph-mon[32769]: pgmap v342: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-26T20:33:17.760 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:17 smithi102 ceph-mon[28094]: pgmap v343: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-26T20:33:17.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:17 smithi130 ceph-mon[32704]: pgmap v343: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-26T20:33:18.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:17 smithi112 ceph-mon[32769]: pgmap v343: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-26T20:33:19.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:19 smithi102 ceph-mon[28094]: pgmap v344: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-26T20:33:20.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:19 smithi112 ceph-mon[32769]: pgmap v344: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-26T20:33:20.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:19 smithi130 ceph-mon[32704]: pgmap v344: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-26T20:33:21.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:21 smithi102 ceph-mon[28094]: pgmap v345: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-26T20:33:22.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:21 smithi112 ceph-mon[32769]: pgmap v345: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-26T20:33:22.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:21 smithi130 ceph-mon[32704]: pgmap v345: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-26T20:33:23.633 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:23 smithi102 ceph-mon[28094]: pgmap v346: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-26T20:33:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.476+0000 7f05bd600700 1 -- 172.21.15.102:0/3557779697 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05b810fa50 msgr2=0x7f05b810fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.476+0000 7f05bd600700 1 --2- 172.21.15.102:0/3557779697 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05b810fa50 0x7f05b810fe30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009990 tx=0x7f05a8009ca0 comp rx=0 tx=0).stop 2024-04-26T20:33:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.476+0000 7f05bd600700 1 -- 172.21.15.102:0/3557779697 shutdown_connections 2024-04-26T20:33:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.476+0000 7f05bd600700 1 --2- 172.21.15.102:0/3557779697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b810a410 0x7f05b8116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.476+0000 7f05bd600700 1 --2- 172.21.15.102:0/3557779697 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05b810fa50 0x7f05b810fe30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.476+0000 7f05bd600700 1 --2- 172.21.15.102:0/3557779697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b8109a50 0x7f05b8109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.476+0000 7f05bd600700 1 -- 172.21.15.102:0/3557779697 >> 172.21.15.102:0/3557779697 conn(0x7f05b8076bf0 msgr2=0x7f05b8108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.476+0000 7f05bd600700 1 -- 172.21.15.102:0/3557779697 shutdown_connections 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.476+0000 7f05bd600700 1 -- 172.21.15.102:0/3557779697 wait complete. 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.477+0000 7f05bd600700 1 Processor -- start 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.477+0000 7f05bd600700 1 -- start start 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.477+0000 7f05bd600700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05b8109a50 0x7f05b81a00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.477+0000 7f05bd600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b810a410 0x7f05b81a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.477+0000 7f05bd600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b810fa50 0x7f05b81a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.477+0000 7f05bd600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f05b8119c30 con 0x7f05b810a410 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.477+0000 7f05bd600700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f05b8119ab0 con 0x7f05b8109a50 2024-04-26T20:33:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.477+0000 7f05bd600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f05b8119db0 con 0x7f05b810fa50 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.478+0000 7f05b7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b810fa50 0x7f05b81a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.478+0000 7f05b6000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b810a410 0x7f05b81a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.478+0000 7f05b7400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b810fa50 0x7f05b81a4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:42782/0 (socket says 172.21.15.102:42782) 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.478+0000 7f05b7400700 1 -- 172.21.15.102:0/345632213 learned_addr learned my addr 172.21.15.102:0/345632213 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.478+0000 7f05b6a00700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05b8109a50 0x7f05b81a00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.478+0000 7f05b7400700 1 -- 172.21.15.102:0/345632213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05b8109a50 msgr2=0x7f05b81a00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.478+0000 7f05b7400700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05b8109a50 0x7f05b81a00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.478+0000 7f05b7400700 1 -- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b810a410 msgr2=0x7f05b81a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.478+0000 7f05b7400700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b810a410 0x7f05b81a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.478+0000 7f05b7400700 1 -- 172.21.15.102:0/345632213 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b81a50d0 con 0x7f05b810fa50 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.479+0000 7f05b7400700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b810fa50 0x7f05b81a4990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f05b000c370 tx=0x7f05b000c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.479+0000 7f05af400700 1 -- 172.21.15.102:0/345632213 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b00092e0 con 0x7f05b810fa50 2024-04-26T20:33:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.479+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b81a5360 con 0x7f05b810fa50 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.479+0000 7f05af400700 1 -- 172.21.15.102:0/345632213 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05b0003a00 con 0x7f05b810fa50 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.479+0000 7f05af400700 1 -- 172.21.15.102:0/345632213 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b00046f0 con 0x7f05b810fa50 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.479+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b81acc40 con 0x7f05b810fa50 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.480+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f05b8063bb0 con 0x7f05b810fa50 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.482+0000 7f05af400700 1 -- 172.21.15.102:0/345632213 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f05b001a020 con 0x7f05b810fa50 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.483+0000 7f05af400700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f059c0a3cd0 0x7f059c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.484+0000 7f05af400700 1 -- 172.21.15.102:0/345632213 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f05b00d0ab0 con 0x7f05b810fa50 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.484+0000 7f05b6a00700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f059c0a3cd0 0x7f059c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.484+0000 7f05af400700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] conn(0x7f059c0ae990 0x7f059c0b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.484+0000 7f05af400700 1 -- 172.21.15.102:0/345632213 --> [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05b0025ea0 con 0x7f059c0ae990 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.484+0000 7f05af400700 1 -- 172.21.15.102:0/345632213 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f05b00d0e60 con 0x7f05b810fa50 2024-04-26T20:33:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.484+0000 7f05b6000700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] conn(0x7f059c0ae990 0x7f059c0b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.485+0000 7f05b6a00700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f059c0a3cd0 0x7f059c0a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f05a80098e0 tx=0x7f05a8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.485+0000 7f05b6000700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] conn(0x7f059c0ae990 0x7f059c0b0dd0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.486+0000 7f05af400700 1 -- 172.21.15.102:0/345632213 <== osd.6 v2:172.21.15.102:6818/879532763 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f05b0025ea0 con 0x7f059c0ae990 2024-04-26T20:33:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.504+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 --> [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f05b8063fb0 con 0x7f059c0ae990 2024-04-26T20:33:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.505+0000 7f05af400700 1 -- 172.21.15.102:0/345632213 <== osd.6 v2:172.21.15.102:6818/879532763 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f05b8063fb0 con 0x7f059c0ae990 2024-04-26T20:33:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.505+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] conn(0x7f059c0ae990 msgr2=0x7f059c0b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.505+0000 7f05bd600700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] conn(0x7f059c0ae990 0x7f059c0b0dd0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.506+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f059c0a3cd0 msgr2=0x7f059c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.506+0000 7f05bd600700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f059c0a3cd0 0x7f059c0a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f05a80098e0 tx=0x7f05a8005960 comp rx=0 tx=0).stop 2024-04-26T20:33:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.506+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b810fa50 msgr2=0x7f05b81a4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.506+0000 7f05bd600700 1 --2- 172.21.15.102:0/345632213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05b810fa50 0x7f05b81a4990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f05b000c370 tx=0x7f05b000c680 comp rx=0 tx=0).stop 2024-04-26T20:33:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.506+0000 7f05b7400700 1 -- 172.21.15.102:0/345632213 reap_dead start 2024-04-26T20:33:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.506+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 shutdown_connections 2024-04-26T20:33:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.506+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 >> 172.21.15.102:0/345632213 conn(0x7f05b8076bf0 msgr2=0x7f05b8077210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.506+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 shutdown_connections 2024-04-26T20:33:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:22.506+0000 7f05bd600700 1 -- 172.21.15.102:0/345632213 wait complete. 2024-04-26T20:33:24.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:23 smithi112 ceph-mon[32769]: pgmap v346: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-26T20:33:24.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:23 smithi130 ceph-mon[32704]: pgmap v346: 129 pgs: 129 active+clean; 635 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-26T20:33:25.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:25 smithi102 ceph-mon[28094]: pgmap v347: 129 pgs: 129 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-26T20:33:26.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:25 smithi112 ceph-mon[32769]: pgmap v347: 129 pgs: 129 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-26T20:33:26.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:25 smithi130 ceph-mon[32704]: pgmap v347: 129 pgs: 129 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-26T20:33:26.604 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:26.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:33:27.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:33:27.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:33:27.168 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:27.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.449+0000 7fef2e600700 1 -- 172.21.15.102:0/3689270495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef2810fa50 msgr2=0x7fef2810fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.449+0000 7fef2e600700 1 --2- 172.21.15.102:0/3689270495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef2810fa50 0x7fef2810fe30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fef180098f0 tx=0x7fef18009c00 comp rx=0 tx=0).stop 2024-04-26T20:33:27.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.449+0000 7fef2e600700 1 -- 172.21.15.102:0/3689270495 shutdown_connections 2024-04-26T20:33:27.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.449+0000 7fef2e600700 1 --2- 172.21.15.102:0/3689270495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef2810a410 0x7fef28116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.449+0000 7fef2e600700 1 --2- 172.21.15.102:0/3689270495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef2810fa50 0x7fef2810fe30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.449+0000 7fef2e600700 1 --2- 172.21.15.102:0/3689270495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef28109a50 0x7fef28109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.449+0000 7fef2e600700 1 -- 172.21.15.102:0/3689270495 >> 172.21.15.102:0/3689270495 conn(0x7fef28076bf0 msgr2=0x7fef28108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:27.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.449+0000 7fef2e600700 1 -- 172.21.15.102:0/3689270495 shutdown_connections 2024-04-26T20:33:27.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.449+0000 7fef2e600700 1 -- 172.21.15.102:0/3689270495 wait complete. 2024-04-26T20:33:27.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.450+0000 7fef2e600700 1 Processor -- start 2024-04-26T20:33:27.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.450+0000 7fef2e600700 1 -- start start 2024-04-26T20:33:27.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.450+0000 7fef2e600700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef28109a50 0x7fef281a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:27.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.450+0000 7fef2e600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef2810a410 0x7fef281a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:27.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.450+0000 7fef2e600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef2810fa50 0x7fef281a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:27.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.450+0000 7fef2e600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fef28119c90 con 0x7fef2810a410 2024-04-26T20:33:27.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.450+0000 7fef2e600700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fef28119b10 con 0x7fef28109a50 2024-04-26T20:33:27.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.450+0000 7fef2e600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fef28119e10 con 0x7fef2810fa50 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef27400700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef28109a50 0x7fef281a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef27400700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef28109a50 0x7fef281a0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:39720/0 (socket says 172.21.15.102:39720) 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef27400700 1 -- 172.21.15.102:0/1610328128 learned_addr learned my addr 172.21.15.102:0/1610328128 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef26a00700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef2810a410 0x7fef281a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef27e00700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef2810fa50 0x7fef281a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef27400700 1 -- 172.21.15.102:0/1610328128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef2810fa50 msgr2=0x7fef281a49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef27400700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef2810fa50 0x7fef281a49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef27400700 1 -- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef2810a410 msgr2=0x7fef281a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef27400700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef2810a410 0x7fef281a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef27400700 1 -- 172.21.15.102:0/1610328128 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef281a5130 con 0x7fef28109a50 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.451+0000 7fef27400700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef28109a50 0x7fef281a0120 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fef18009230 tx=0x7fef1800fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.452+0000 7fef1fe00700 1 -- 172.21.15.102:0/1610328128 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef180050c0 con 0x7fef28109a50 2024-04-26T20:33:27.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.452+0000 7fef1fe00700 1 -- 172.21.15.102:0/1610328128 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef1800b560 con 0x7fef28109a50 2024-04-26T20:33:27.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.452+0000 7fef1fe00700 1 -- 172.21.15.102:0/1610328128 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef18003a00 con 0x7fef28109a50 2024-04-26T20:33:27.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.452+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef281a53c0 con 0x7fef28109a50 2024-04-26T20:33:27.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.452+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef281acca0 con 0x7fef28109a50 2024-04-26T20:33:27.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.453+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fef281a5850 con 0x7fef28109a50 2024-04-26T20:33:27.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.455+0000 7fef1fe00700 1 -- 172.21.15.102:0/1610328128 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fef1800b710 con 0x7fef28109a50 2024-04-26T20:33:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.456+0000 7fef1fe00700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fef0c0a3cd0 0x7fef0c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.456+0000 7fef1fe00700 1 -- 172.21.15.102:0/1610328128 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fef180d1b30 con 0x7fef28109a50 2024-04-26T20:33:27.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.456+0000 7fef26a00700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fef0c0a3cd0 0x7fef0c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.457+0000 7fef1fe00700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] conn(0x7fef0c0ae860 0x7fef0c0b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.457+0000 7fef1fe00700 1 -- 172.21.15.102:0/1610328128 --> [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef0c0b1370 con 0x7fef0c0ae860 2024-04-26T20:33:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.457+0000 7fef1fe00700 1 -- 172.21.15.102:0/1610328128 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fef180d1ee0 con 0x7fef28109a50 2024-04-26T20:33:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.457+0000 7fef27e00700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] conn(0x7fef0c0ae860 0x7fef0c0b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.457+0000 7fef26a00700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fef0c0a3cd0 0x7fef0c0a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fef28078b60 tx=0x7fef14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:27.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.458+0000 7fef27e00700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] conn(0x7fef0c0ae860 0x7fef0c0b0ca0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:27.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.459+0000 7fef1fe00700 1 -- 172.21.15.102:0/1610328128 <== osd.0 v2:172.21.15.102:6802/1690985624 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fef0c0b1370 con 0x7fef0c0ae860 2024-04-26T20:33:27.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.476+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 --> [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fef28063e40 con 0x7fef0c0ae860 2024-04-26T20:33:27.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.477+0000 7fef1fe00700 1 -- 172.21.15.102:0/1610328128 <== osd.0 v2:172.21.15.102:6802/1690985624 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fef28063e40 con 0x7fef0c0ae860 2024-04-26T20:33:27.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.477+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] conn(0x7fef0c0ae860 msgr2=0x7fef0c0b0ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.477+0000 7fef2e600700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] conn(0x7fef0c0ae860 0x7fef0c0b0ca0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.477+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fef0c0a3cd0 msgr2=0x7fef0c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.477+0000 7fef2e600700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fef0c0a3cd0 0x7fef0c0a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fef28078b60 tx=0x7fef14009260 comp rx=0 tx=0).stop 2024-04-26T20:33:27.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.477+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef28109a50 msgr2=0x7fef281a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.477+0000 7fef2e600700 1 --2- 172.21.15.102:0/1610328128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef28109a50 0x7fef281a0120 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fef18009230 tx=0x7fef1800fa70 comp rx=0 tx=0).stop 2024-04-26T20:33:27.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.477+0000 7fef27e00700 1 -- 172.21.15.102:0/1610328128 reap_dead start 2024-04-26T20:33:27.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.478+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 shutdown_connections 2024-04-26T20:33:27.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.478+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 >> 172.21.15.102:0/1610328128 conn(0x7fef28076bf0 msgr2=0x7fef28077210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:27.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.478+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 shutdown_connections 2024-04-26T20:33:27.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.478+0000 7fef2e600700 1 -- 172.21.15.102:0/1610328128 wait complete. 2024-04-26T20:33:27.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.696+0000 7f10c0200700 1 -- 172.21.15.102:0/2606570655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10b8103470 msgr2=0x7f10b81038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.696+0000 7f10c0200700 1 --2- 172.21.15.102:0/2606570655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10b8103470 0x7f10b81038f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009990 tx=0x7f10a4009ca0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.696+0000 7f10c0200700 1 -- 172.21.15.102:0/2606570655 shutdown_connections 2024-04-26T20:33:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.696+0000 7f10c0200700 1 --2- 172.21.15.102:0/2606570655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b8103e30 0x7f10b8110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.696+0000 7f10c0200700 1 --2- 172.21.15.102:0/2606570655 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10b81094c0 0x7f10b81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.696+0000 7f10c0200700 1 --2- 172.21.15.102:0/2606570655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10b8103470 0x7f10b81038f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.696+0000 7f10c0200700 1 -- 172.21.15.102:0/2606570655 >> 172.21.15.102:0/2606570655 conn(0x7f10b807a580 msgr2=0x7f10b807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.696+0000 7f10c0200700 1 -- 172.21.15.102:0/2606570655 shutdown_connections 2024-04-26T20:33:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.696+0000 7f10c0200700 1 -- 172.21.15.102:0/2606570655 wait complete. 2024-04-26T20:33:27.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.697+0000 7f10c0200700 1 Processor -- start 2024-04-26T20:33:27.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.697+0000 7f10c0200700 1 -- start start 2024-04-26T20:33:27.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.697+0000 7f10c0200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10b8103470 0x7f10b819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:27.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.697+0000 7f10c0200700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10b8103e30 0x7f10b819c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.697+0000 7f10c0200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b81094c0 0x7f10b81a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.697+0000 7f10c0200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f10b8113610 con 0x7f10b81094c0 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.697+0000 7f10c0200700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f10b8113490 con 0x7f10b8103e30 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.697+0000 7f10c0200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f10b8113790 con 0x7f10b8103470 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10bd800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10b8103470 0x7f10b819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10bd800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10b8103470 0x7f10b819bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:44280/0 (socket says 172.21.15.102:44280) 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10bd800700 1 -- 172.21.15.102:0/4278796473 learned_addr learned my addr 172.21.15.102:0/4278796473 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10be200700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b81094c0 0x7f10b81a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10bce00700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10b8103e30 0x7f10b819c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10bd800700 1 -- 172.21.15.102:0/4278796473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10b8103e30 msgr2=0x7f10b819c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10bd800700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10b8103e30 0x7f10b819c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10bd800700 1 -- 172.21.15.102:0/4278796473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b81094c0 msgr2=0x7f10b81a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10bd800700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b81094c0 0x7f10b81a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10bd800700 1 -- 172.21.15.102:0/4278796473 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b81a0d50 con 0x7f10b8103470 2024-04-26T20:33:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10be200700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b81094c0 0x7f10b81a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.698+0000 7f10bce00700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f10b8103e30 0x7f10b819c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.699+0000 7f10bd800700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10b8103470 0x7f10b819bd40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00e9a0 tx=0x7f10ac00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.699+0000 7f10b6000700 1 -- 172.21.15.102:0/4278796473 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac00ca10 con 0x7f10b8103470 2024-04-26T20:33:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.699+0000 7f10b6000700 1 -- 172.21.15.102:0/4278796473 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10ac00cbb0 con 0x7f10b8103470 2024-04-26T20:33:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.699+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b81a1040 con 0x7f10b8103470 2024-04-26T20:33:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.699+0000 7f10b6000700 1 -- 172.21.15.102:0/4278796473 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac018680 con 0x7f10b8103470 2024-04-26T20:33:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.699+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b81a88c0 con 0x7f10b8103470 2024-04-26T20:33:27.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.700+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f10b8063bb0 con 0x7f10b8103470 2024-04-26T20:33:27.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.702+0000 7f10b6000700 1 -- 172.21.15.102:0/4278796473 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f10ac010430 con 0x7f10b8103470 2024-04-26T20:33:27.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.703+0000 7f10b6000700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10a00a3cd0 0x7f10a00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:27.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.703+0000 7f10b6000700 1 -- 172.21.15.102:0/4278796473 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f10ac0d0910 con 0x7f10b8103470 2024-04-26T20:33:27.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.703+0000 7f10bce00700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10a00a3cd0 0x7f10a00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:27.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.703+0000 7f10b6000700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] conn(0x7f10a00ae990 0x7f10a00b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:27.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.703+0000 7f10b6000700 1 -- 172.21.15.102:0/4278796473 --> [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10a00b14a0 con 0x7f10a00ae990 2024-04-26T20:33:27.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.703+0000 7f10b6000700 1 -- 172.21.15.102:0/4278796473 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f10ac0d52a0 con 0x7f10b8103470 2024-04-26T20:33:27.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.704+0000 7f10be200700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] conn(0x7f10a00ae990 0x7f10a00b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:27.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.704+0000 7f10bce00700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10a00a3cd0 0x7f10a00a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f10b807ae80 tx=0x7f10a400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:27.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.704+0000 7f10be200700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] conn(0x7f10a00ae990 0x7f10a00b0dd0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:27.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.705+0000 7f10b6000700 1 -- 172.21.15.102:0/4278796473 <== osd.4 v2:172.21.15.112:6808/926818023 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10a00b14a0 con 0x7f10a00ae990 2024-04-26T20:33:27.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.723+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 --> [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f10b8064030 con 0x7f10a00ae990 2024-04-26T20:33:27.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.723+0000 7f10b6000700 1 -- 172.21.15.102:0/4278796473 <== osd.4 v2:172.21.15.112:6808/926818023 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f10b8064030 con 0x7f10a00ae990 2024-04-26T20:33:27.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 >> [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] conn(0x7f10a00ae990 msgr2=0x7f10a00b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10c0200700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] conn(0x7f10a00ae990 0x7f10a00b0dd0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10a00a3cd0 msgr2=0x7f10a00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10c0200700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f10a00a3cd0 0x7f10a00a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f10b807ae80 tx=0x7f10a400b260 comp rx=0 tx=0).stop 2024-04-26T20:33:27.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10b8103470 msgr2=0x7f10b819bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:27.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10c0200700 1 --2- 172.21.15.102:0/4278796473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10b8103470 0x7f10b819bd40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00e9a0 tx=0x7f10ac00ecb0 comp rx=0 tx=0).stop 2024-04-26T20:33:27.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10be200700 1 -- 172.21.15.102:0/4278796473 reap_dead start 2024-04-26T20:33:27.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 shutdown_connections 2024-04-26T20:33:27.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 >> 172.21.15.102:0/4278796473 conn(0x7f10b807a580 msgr2=0x7f10b810e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:27.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 shutdown_connections 2024-04-26T20:33:27.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:27.724+0000 7f10c0200700 1 -- 172.21.15.102:0/4278796473 wait complete. 2024-04-26T20:33:27.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:27 smithi130 ceph-mon[32704]: pgmap v348: 129 pgs: 129 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-26T20:33:27.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:27 smithi102 ceph-mon[28094]: pgmap v348: 129 pgs: 129 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-26T20:33:28.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:27 smithi112 ceph-mon[32769]: pgmap v348: 129 pgs: 129 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-26T20:33:28.751 INFO:teuthology.orchestra.run.smithi102.stdout:197568495669 2024-04-26T20:33:28.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T20:33:29.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-26T20:33:29.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-26T20:33:29.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-26T20:33:29.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-26T20:33:29.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-26T20:33:29.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-04-26T20:33:29.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-04-26T20:33:29.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]: dispatch 2024-04-26T20:33:29.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-26T20:33:29.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-26T20:33:29.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:29 smithi102 ceph-mon[28094]: pgmap v349: 129 pgs: 129 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-04-26T20:33:29.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-04-26T20:33:29.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]: dispatch 2024-04-26T20:33:29.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-26T20:33:29.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-26T20:33:29.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:29 smithi112 ceph-mon[32769]: pgmap v349: 129 pgs: 129 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-04-26T20:33:29.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-04-26T20:33:29.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]: dispatch 2024-04-26T20:33:29.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-26T20:33:29.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-26T20:33:29.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:29 smithi130 ceph-mon[32704]: pgmap v349: 129 pgs: 129 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-26T20:33:30.594 INFO:teuthology.orchestra.run.smithi102.stdout:34359738454 2024-04-26T20:33:30.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T20:33:30.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-26T20:33:30.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-26T20:33:30.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-26T20:33:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-26T20:33:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-26T20:33:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-04-26T20:33:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-04-26T20:33:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]': finished 2024-04-26T20:33:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-26T20:33:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-26T20:33:30.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:30 smithi102 ceph-mon[28094]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T20:33:30.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-26T20:33:30.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-26T20:33:30.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-26T20:33:30.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-26T20:33:30.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-26T20:33:30.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-04-26T20:33:30.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-04-26T20:33:30.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]': finished 2024-04-26T20:33:30.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-26T20:33:30.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-26T20:33:30.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:30 smithi112 ceph-mon[32769]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T20:33:30.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-26T20:33:30.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-26T20:33:30.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-26T20:33:30.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-26T20:33:30.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-26T20:33:30.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-04-26T20:33:30.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-04-26T20:33:30.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]': finished 2024-04-26T20:33:30.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-26T20:33:30.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-26T20:33:30.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:30 smithi130 ceph-mon[32704]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T20:33:31.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.157+0000 7fd58ee00700 1 -- 172.21.15.102:0/3114638267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd58806b670 msgr2=0x7fd58806bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:31.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.157+0000 7fd58ee00700 1 --2- 172.21.15.102:0/3114638267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd58806b670 0x7fd58806bad0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd574009990 tx=0x7fd574009ca0 comp rx=0 tx=0).stop 2024-04-26T20:33:31.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.157+0000 7fd58ee00700 1 -- 172.21.15.102:0/3114638267 shutdown_connections 2024-04-26T20:33:31.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.157+0000 7fd58ee00700 1 --2- 172.21.15.102:0/3114638267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd588105e60 0x7fd5881062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:31.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.157+0000 7fd58ee00700 1 --2- 172.21.15.102:0/3114638267 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd588108350 0x7fd58806b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:31.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.157+0000 7fd58ee00700 1 --2- 172.21.15.102:0/3114638267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd58806b670 0x7fd58806bad0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.157+0000 7fd58ee00700 1 -- 172.21.15.102:0/3114638267 >> 172.21.15.102:0/3114638267 conn(0x7fd5880792d0 msgr2=0x7fd5880796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.158+0000 7fd58ee00700 1 -- 172.21.15.102:0/3114638267 shutdown_connections 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.158+0000 7fd58ee00700 1 -- 172.21.15.102:0/3114638267 wait complete. 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.159+0000 7fd58ee00700 1 Processor -- start 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.159+0000 7fd58ee00700 1 -- start start 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd58ee00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd58806b670 0x7fd5881a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd58ee00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd588105e60 0x7fd5881a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd58ee00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd588108350 0x7fd5881a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd58ee00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd588119b90 con 0x7fd588105e60 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd58ee00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd588119a10 con 0x7fd588108350 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd58ee00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd588119d10 con 0x7fd58806b670 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd587400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd588105e60 0x7fd5881a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd587400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd588105e60 0x7fd5881a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39754/0 (socket says 172.21.15.102:39754) 2024-04-26T20:33:31.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd587400700 1 -- 172.21.15.102:0/3112808900 learned_addr learned my addr 172.21.15.102:0/3112808900 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd58ce00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd588108350 0x7fd5881a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.160+0000 7fd587e00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd58806b670 0x7fd5881a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd587400700 1 -- 172.21.15.102:0/3112808900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd58806b670 msgr2=0x7fd5881a00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd587400700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd58806b670 0x7fd5881a00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd587400700 1 -- 172.21.15.102:0/3112808900 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd588108350 msgr2=0x7fd5881a49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd587400700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd588108350 0x7fd5881a49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd587400700 1 -- 172.21.15.102:0/3112808900 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5881a50f0 con 0x7fd588105e60 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd58ce00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd588108350 0x7fd5881a49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd587400700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd588105e60 0x7fd5881a0620 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd5740057f0 tx=0x7fd57400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd584c00700 1 -- 172.21.15.102:0/3112808900 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd574026070 con 0x7fd588105e60 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5881a5380 con 0x7fd588105e60 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd584c00700 1 -- 172.21.15.102:0/3112808900 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd574021410 con 0x7fd588105e60 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.161+0000 7fd584c00700 1 -- 172.21.15.102:0/3112808900 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd574017650 con 0x7fd588105e60 2024-04-26T20:33:31.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.162+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5881acc60 con 0x7fd588105e60 2024-04-26T20:33:31.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.163+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd5881048a0 con 0x7fd588105e60 2024-04-26T20:33:31.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.166+0000 7fd584c00700 1 -- 172.21.15.102:0/3112808900 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fd574029070 con 0x7fd588105e60 2024-04-26T20:33:31.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.167+0000 7fd584c00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd56c0a3d50 0x7fd56c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:31.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.167+0000 7fd584c00700 1 -- 172.21.15.102:0/3112808900 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd5740d18c0 con 0x7fd588105e60 2024-04-26T20:33:31.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.168+0000 7fd587e00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd56c0a3d50 0x7fd56c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.168+0000 7fd584c00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] conn(0x7fd56c0aed80 0x7fd56c0b1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.168+0000 7fd584c00700 1 -- 172.21.15.102:0/3112808900 --> [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd56c0b18f0 con 0x7fd56c0aed80 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.168+0000 7fd584c00700 1 -- 172.21.15.102:0/3112808900 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fd5740d1c70 con 0x7fd588105e60 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.168+0000 7fd58ce00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] conn(0x7fd56c0aed80 0x7fd56c0b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.168+0000 7fd587e00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd56c0a3d50 0x7fd56c0a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd578009be0 tx=0x7fd578009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.169+0000 7fd58ce00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] conn(0x7fd56c0aed80 0x7fd56c0b1220 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.170+0000 7fd584c00700 1 -- 172.21.15.102:0/3112808900 <== osd.5 v2:172.21.15.130:6808/1600420377 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd56c0b18f0 con 0x7fd56c0aed80 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.188+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 --> [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd588102940 con 0x7fd56c0aed80 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.188+0000 7fd584c00700 1 -- 172.21.15.102:0/3112808900 <== osd.5 v2:172.21.15.130:6808/1600420377 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd588102940 con 0x7fd56c0aed80 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.189+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 >> [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] conn(0x7fd56c0aed80 msgr2=0x7fd56c0b1220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.189+0000 7fd58ee00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] conn(0x7fd56c0aed80 0x7fd56c0b1220 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.189+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd56c0a3d50 msgr2=0x7fd56c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.189+0000 7fd58ee00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd56c0a3d50 0x7fd56c0a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd578009be0 tx=0x7fd578009260 comp rx=0 tx=0).stop 2024-04-26T20:33:31.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.189+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd588105e60 msgr2=0x7fd5881a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:31.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.189+0000 7fd58ee00700 1 --2- 172.21.15.102:0/3112808900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd588105e60 0x7fd5881a0620 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd5740057f0 tx=0x7fd57400bfd0 comp rx=0 tx=0).stop 2024-04-26T20:33:31.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.189+0000 7fd58ce00700 1 -- 172.21.15.102:0/3112808900 reap_dead start 2024-04-26T20:33:31.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.189+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 shutdown_connections 2024-04-26T20:33:31.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.189+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 >> 172.21.15.102:0/3112808900 conn(0x7fd5880792d0 msgr2=0x7fd588102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:31.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.190+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 shutdown_connections 2024-04-26T20:33:31.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:31.190+0000 7fd58ee00700 1 -- 172.21.15.102:0/3112808900 wait complete. 2024-04-26T20:33:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:31 smithi102 ceph-mon[28094]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T20:33:31.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:31 smithi102 ceph-mon[28094]: pgmap v352: 129 pgs: 4 peering, 125 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 383 B/s wr, 11 op/s 2024-04-26T20:33:31.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:31 smithi112 ceph-mon[32769]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T20:33:31.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:31 smithi112 ceph-mon[32769]: pgmap v352: 129 pgs: 4 peering, 125 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 383 B/s wr, 11 op/s 2024-04-26T20:33:31.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:31 smithi130 ceph-mon[32704]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T20:33:31.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:31 smithi130 ceph-mon[32704]: pgmap v352: 129 pgs: 4 peering, 125 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 383 B/s wr, 11 op/s 2024-04-26T20:33:32.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.518+0000 7f408bc00700 1 -- 172.21.15.102:0/1925289776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4084103e30 msgr2=0x7f40841102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:32.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.518+0000 7f408bc00700 1 --2- 172.21.15.102:0/1925289776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4084103e30 0x7f40841102c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f407c009f90 tx=0x7f407c00b330 comp rx=0 tx=0).stop 2024-04-26T20:33:32.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.518+0000 7f408bc00700 1 -- 172.21.15.102:0/1925289776 shutdown_connections 2024-04-26T20:33:32.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.518+0000 7f408bc00700 1 --2- 172.21.15.102:0/1925289776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4084103e30 0x7f40841102c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:32.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.518+0000 7f408bc00700 1 --2- 172.21.15.102:0/1925289776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f40841094f0 0x7f40841098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:32.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.518+0000 7f408bc00700 1 --2- 172.21.15.102:0/1925289776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4084103470 0x7f40841038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:32.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.518+0000 7f408bc00700 1 -- 172.21.15.102:0/1925289776 >> 172.21.15.102:0/1925289776 conn(0x7f408407a5d0 msgr2=0x7f408407a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:32.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.518+0000 7f408bc00700 1 -- 172.21.15.102:0/1925289776 shutdown_connections 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.518+0000 7f408bc00700 1 -- 172.21.15.102:0/1925289776 wait complete. 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.519+0000 7f408bc00700 1 Processor -- start 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.519+0000 7f408bc00700 1 -- start start 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f408bc00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4084103470 0x7f40841a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f408bc00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4084103e30 0x7f40841a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f408bc00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40841094f0 0x7f408419bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f408bc00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4084113640 con 0x7f40841094f0 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f408bc00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f40841134c0 con 0x7f4084103470 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f408bc00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f40841137c0 con 0x7f4084103e30 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f4089c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40841094f0 0x7f408419bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f4083e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4084103e30 0x7f40841a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f4089c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40841094f0 0x7f408419bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39770/0 (socket says 172.21.15.102:39770) 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f4089c00700 1 -- 172.21.15.102:0/2151934764 learned_addr learned my addr 172.21.15.102:0/2151934764 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.520+0000 7f4089200700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4084103470 0x7f40841a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:32.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4083e00700 1 -- 172.21.15.102:0/2151934764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4084103470 msgr2=0x7f40841a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4083e00700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4084103470 0x7f40841a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4083e00700 1 -- 172.21.15.102:0/2151934764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40841094f0 msgr2=0x7f408419bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4083e00700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40841094f0 0x7f408419bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4083e00700 1 -- 172.21.15.102:0/2151934764 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408419c230 con 0x7f4084103e30 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4089c00700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40841094f0 0x7f408419bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4089200700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4084103470 0x7f40841a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4083e00700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4084103e30 0x7f40841a1e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9f0 tx=0x7f407000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4081600700 1 -- 172.21.15.102:0/2151934764 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407000c9c0 con 0x7f4084103e30 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4081600700 1 -- 172.21.15.102:0/2151934764 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f407000cb60 con 0x7f4084103e30 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408419c520 con 0x7f4084103e30 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.521+0000 7f4081600700 1 -- 172.21.15.102:0/2151934764 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070018680 con 0x7f4084103e30 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.522+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40841a87e0 con 0x7f4084103e30 2024-04-26T20:33:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.523+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4084110310 con 0x7f4084103e30 2024-04-26T20:33:32.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.525+0000 7f4081600700 1 -- 172.21.15.102:0/2151934764 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f4070010430 con 0x7f4084103e30 2024-04-26T20:33:32.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.527+0000 7f4081600700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f406c0a3d50 0x7f406c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:32.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.527+0000 7f4081600700 1 -- 172.21.15.102:0/2151934764 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f40700d0a30 con 0x7f4084103e30 2024-04-26T20:33:32.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.528+0000 7f4089200700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f406c0a3d50 0x7f406c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:32.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.528+0000 7f4081600700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] conn(0x7f406c0aeeb0 0x7f406c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:32.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.528+0000 7f4081600700 1 -- 172.21.15.102:0/2151934764 --> [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4070015e80 con 0x7f406c0aeeb0 2024-04-26T20:33:32.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.528+0000 7f4081600700 1 -- 172.21.15.102:0/2151934764 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f40700d52a0 con 0x7f4084103e30 2024-04-26T20:33:32.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.528+0000 7f4089c00700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] conn(0x7f406c0aeeb0 0x7f406c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:32.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.529+0000 7f4089200700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f406c0a3d50 0x7f406c0a6210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4078005fd0 tx=0x7f4078009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:32.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.529+0000 7f4089c00700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] conn(0x7f406c0aeeb0 0x7f406c0b1350 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.530+0000 7f4081600700 1 -- 172.21.15.102:0/2151934764 <== osd.8 v2:172.21.15.130:6816/3061190043 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4070015e80 con 0x7f406c0aeeb0 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.550+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 --> [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4084104b90 con 0x7f406c0aeeb0 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.551+0000 7f4081600700 1 -- 172.21.15.102:0/2151934764 <== osd.8 v2:172.21.15.130:6816/3061190043 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4084104b90 con 0x7f406c0aeeb0 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 >> [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] conn(0x7f406c0aeeb0 msgr2=0x7f406c0b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f408bc00700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] conn(0x7f406c0aeeb0 0x7f406c0b1350 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f406c0a3d50 msgr2=0x7f406c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f408bc00700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f406c0a3d50 0x7f406c0a6210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4078005fd0 tx=0x7f4078009310 comp rx=0 tx=0).stop 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4084103e30 msgr2=0x7f40841a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f408bc00700 1 --2- 172.21.15.102:0/2151934764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4084103e30 0x7f40841a1e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9f0 tx=0x7f407000ed00 comp rx=0 tx=0).stop 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f4089c00700 1 -- 172.21.15.102:0/2151934764 reap_dead start 2024-04-26T20:33:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 shutdown_connections 2024-04-26T20:33:32.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 >> 172.21.15.102:0/2151934764 conn(0x7f408407a5d0 msgr2=0x7f408410e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:32.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 shutdown_connections 2024-04-26T20:33:32.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:32.552+0000 7f408bc00700 1 -- 172.21.15.102:0/2151934764 wait complete. 2024-04-26T20:33:32.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:32 smithi102 ceph-mon[28094]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-26T20:33:33.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:32 smithi112 ceph-mon[32769]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-26T20:33:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:32 smithi130 ceph-mon[32704]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-26T20:33:33.897 INFO:teuthology.orchestra.run.smithi102.stdout:141733920835 2024-04-26T20:33:33.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T20:33:33.952 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:33 smithi102 ceph-mon[28094]: pgmap v353: 129 pgs: 4 peering, 125 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 77 B/s, 0 objects/s recovering 2024-04-26T20:33:34.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:33 smithi112 ceph-mon[32769]: pgmap v353: 129 pgs: 4 peering, 125 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 77 B/s, 0 objects/s recovering 2024-04-26T20:33:34.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:33 smithi130 ceph-mon[32704]: pgmap v353: 129 pgs: 4 peering, 125 active+clean; 637 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 77 B/s, 0 objects/s recovering 2024-04-26T20:33:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.244+0000 7fe89e000700 1 -- 172.21.15.102:0/2331372281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe8981094a0 msgr2=0x7fe898109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.244+0000 7fe89e000700 1 --2- 172.21.15.102:0/2331372281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe8981094a0 0x7fe898109880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe8880098f0 tx=0x7fe888009c00 comp rx=0 tx=0).stop 2024-04-26T20:33:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.244+0000 7fe89e000700 1 -- 172.21.15.102:0/2331372281 shutdown_connections 2024-04-26T20:33:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.244+0000 7fe89e000700 1 --2- 172.21.15.102:0/2331372281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe89806bad0 0x7fe8981102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.244+0000 7fe89e000700 1 --2- 172.21.15.102:0/2331372281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe8981094a0 0x7fe898109880 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.244+0000 7fe89e000700 1 --2- 172.21.15.102:0/2331372281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe89806b130 0x7fe89806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.244+0000 7fe89e000700 1 -- 172.21.15.102:0/2331372281 >> 172.21.15.102:0/2331372281 conn(0x7fe8980792d0 msgr2=0x7fe8980796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.244+0000 7fe89e000700 1 -- 172.21.15.102:0/2331372281 shutdown_connections 2024-04-26T20:33:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.245+0000 7fe89e000700 1 -- 172.21.15.102:0/2331372281 wait complete. 2024-04-26T20:33:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.246+0000 7fe89e000700 1 Processor -- start 2024-04-26T20:33:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.246+0000 7fe89e000700 1 -- start start 2024-04-26T20:33:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe89e000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe89806b130 0x7fe89819bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe89e000700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe89806bad0 0x7fe89819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe89e000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8981a0880 0x7fe8981a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe89e000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe8981135e0 con 0x7fe8981a0880 2024-04-26T20:33:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe89e000700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe898113460 con 0x7fe89806bad0 2024-04-26T20:33:34.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe89e000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe898113760 con 0x7fe89806b130 2024-04-26T20:33:34.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe897400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe89806b130 0x7fe89819bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe897400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe89806b130 0x7fe89819bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:44334/0 (socket says 172.21.15.102:44334) 2024-04-26T20:33:34.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe897400700 1 -- 172.21.15.102:0/17056636 learned_addr learned my addr 172.21.15.102:0/17056636 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:34.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe896a00700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe89806bad0 0x7fe89819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.247+0000 7fe897e00700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8981a0880 0x7fe8981a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.248+0000 7fe896a00700 1 -- 172.21.15.102:0/17056636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe89806b130 msgr2=0x7fe89819bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.248+0000 7fe896a00700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe89806b130 0x7fe89819bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.248+0000 7fe896a00700 1 -- 172.21.15.102:0/17056636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8981a0880 msgr2=0x7fe8981a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.248+0000 7fe896a00700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8981a0880 0x7fe8981a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.248+0000 7fe896a00700 1 -- 172.21.15.102:0/17056636 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8981a13a0 con 0x7fe89806bad0 2024-04-26T20:33:34.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.248+0000 7fe897400700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe89806b130 0x7fe89819bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:34.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.248+0000 7fe897e00700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8981a0880 0x7fe8981a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:34.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.249+0000 7fe896a00700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe89806bad0 0x7fe89819c4f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe88400e9c0 tx=0x7fe88400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:34.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.249+0000 7fe88fe00700 1 -- 172.21.15.102:0/17056636 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88400c9c0 con 0x7fe89806bad0 2024-04-26T20:33:34.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.249+0000 7fe88fe00700 1 -- 172.21.15.102:0/17056636 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe88400cb60 con 0x7fe89806bad0 2024-04-26T20:33:34.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.249+0000 7fe88fe00700 1 -- 172.21.15.102:0/17056636 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884018420 con 0x7fe89806bad0 2024-04-26T20:33:34.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.249+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8981a1690 con 0x7fe89806bad0 2024-04-26T20:33:34.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.249+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8981a8db0 con 0x7fe89806bad0 2024-04-26T20:33:34.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.250+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe898100c50 con 0x7fe89806bad0 2024-04-26T20:33:34.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.253+0000 7fe88fe00700 1 -- 172.21.15.102:0/17056636 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fe884010430 con 0x7fe89806bad0 2024-04-26T20:33:34.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.254+0000 7fe88fe00700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe87c0a3cd0 0x7fe87c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.255+0000 7fe88fe00700 1 -- 172.21.15.102:0/17056636 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe8840d0b90 con 0x7fe89806bad0 2024-04-26T20:33:34.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.255+0000 7fe88fe00700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] conn(0x7fe87c0aee30 0x7fe87c0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.255+0000 7fe88fe00700 1 -- 172.21.15.102:0/17056636 --> [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe87c0b19a0 con 0x7fe87c0aee30 2024-04-26T20:33:34.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.255+0000 7fe88fe00700 1 -- 172.21.15.102:0/17056636 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fe8840d0f40 con 0x7fe89806bad0 2024-04-26T20:33:34.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.255+0000 7fe897400700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe87c0a3cd0 0x7fe87c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.256+0000 7fe897e00700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] conn(0x7fe87c0aee30 0x7fe87c0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.256+0000 7fe897e00700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] conn(0x7fe87c0aee30 0x7fe87c0b12d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:34.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.257+0000 7fe897400700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe87c0a3cd0 0x7fe87c0a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe888009810 tx=0x7fe888019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:34.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.257+0000 7fe88fe00700 1 -- 172.21.15.102:0/17056636 <== osd.11 v2:172.21.15.130:6824/822414281 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe87c0b19a0 con 0x7fe87c0aee30 2024-04-26T20:33:34.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.276+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 --> [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe898104b50 con 0x7fe87c0aee30 2024-04-26T20:33:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.277+0000 7fe88fe00700 1 -- 172.21.15.102:0/17056636 <== osd.11 v2:172.21.15.130:6824/822414281 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe898104b50 con 0x7fe87c0aee30 2024-04-26T20:33:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.277+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 >> [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] conn(0x7fe87c0aee30 msgr2=0x7fe87c0b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.277+0000 7fe89e000700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] conn(0x7fe87c0aee30 0x7fe87c0b12d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.278+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe87c0a3cd0 msgr2=0x7fe87c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.278+0000 7fe89e000700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe87c0a3cd0 0x7fe87c0a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe888009810 tx=0x7fe888019040 comp rx=0 tx=0).stop 2024-04-26T20:33:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.278+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe89806bad0 msgr2=0x7fe89819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.278+0000 7fe89e000700 1 --2- 172.21.15.102:0/17056636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe89806bad0 0x7fe89819c4f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe88400e9c0 tx=0x7fe88400ecd0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.278+0000 7fe897e00700 1 -- 172.21.15.102:0/17056636 reap_dead start 2024-04-26T20:33:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.278+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 shutdown_connections 2024-04-26T20:33:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.278+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 >> 172.21.15.102:0/17056636 conn(0x7fe8980792d0 msgr2=0x7fe898100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.278+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 shutdown_connections 2024-04-26T20:33:34.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.278+0000 7fe89e000700 1 -- 172.21.15.102:0/17056636 wait complete. 2024-04-26T20:33:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.153+0000 7f2bb5000700 1 -- 172.21.15.102:0/1985511349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb01145f0 msgr2=0x7f2bb01169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.153+0000 7f2bb5000700 1 --2- 172.21.15.102:0/1985511349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb01145f0 0x7f2bb01169e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800b0a0 tx=0x7f2ba800b3b0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.153+0000 7f2bb5000700 1 -- 172.21.15.102:0/1985511349 shutdown_connections 2024-04-26T20:33:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.153+0000 7f2bb5000700 1 --2- 172.21.15.102:0/1985511349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb01145f0 0x7f2bb01169e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.153+0000 7f2bb5000700 1 --2- 172.21.15.102:0/1985511349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb0075650 0x7f2bb0075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.153+0000 7f2bb5000700 1 --2- 172.21.15.102:0/1985511349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2bb0075f70 0x7f2bb01140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.153+0000 7f2bb5000700 1 -- 172.21.15.102:0/1985511349 >> 172.21.15.102:0/1985511349 conn(0x7f2bb007a580 msgr2=0x7f2bb007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.153+0000 7f2bb5000700 1 -- 172.21.15.102:0/1985511349 shutdown_connections 2024-04-26T20:33:34.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.153+0000 7f2bb5000700 1 -- 172.21.15.102:0/1985511349 wait complete. 2024-04-26T20:33:34.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.154+0000 7f2bb5000700 1 Processor -- start 2024-04-26T20:33:34.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.154+0000 7f2bb5000700 1 -- start start 2024-04-26T20:33:34.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bb5000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb0075650 0x7f2bb01a00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bb5000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb0075f70 0x7f2bb01a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bb5000700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2bb01145f0 0x7f2bb01a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bb5000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0119bd0 con 0x7f2bb0075650 2024-04-26T20:33:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bb5000700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0119a50 con 0x7f2bb01145f0 2024-04-26T20:33:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bb5000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0119d50 con 0x7f2bb0075f70 2024-04-26T20:33:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bade00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb0075650 0x7f2bb01a00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bad400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb0075f70 0x7f2bb01a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bade00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb0075650 0x7f2bb01a00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39792/0 (socket says 172.21.15.102:39792) 2024-04-26T20:33:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bade00700 1 -- 172.21.15.102:0/1996229943 learned_addr learned my addr 172.21.15.102:0/1996229943 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bad400700 1 -- 172.21.15.102:0/1996229943 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2bb01145f0 msgr2=0x7f2bb01a4990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bad400700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2bb01145f0 0x7f2bb01a4990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bad400700 1 -- 172.21.15.102:0/1996229943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb0075650 msgr2=0x7f2bb01a00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bad400700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb0075650 0x7f2bb01a00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.155+0000 7f2bad400700 1 -- 172.21.15.102:0/1996229943 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb01a50d0 con 0x7f2bb0075f70 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.156+0000 7f2bad400700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb0075f70 0x7f2bb01a0600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c00e9f0 tx=0x7f2b9c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.156+0000 7f2ba6a00700 1 -- 172.21.15.102:0/1996229943 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c00cbd0 con 0x7f2bb0075f70 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.156+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb01a53c0 con 0x7f2bb0075f70 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.156+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb01acc40 con 0x7f2bb0075f70 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.156+0000 7f2ba6a00700 1 -- 172.21.15.102:0/1996229943 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b9c00cd70 con 0x7f2bb0075f70 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.156+0000 7f2ba6a00700 1 -- 172.21.15.102:0/1996229943 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c018710 con 0x7f2bb0075f70 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.157+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2b90001030 con 0x7f2bb0075f70 2024-04-26T20:33:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.162+0000 7f2ba6a00700 1 -- 172.21.15.102:0/1996229943 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2b9c010970 con 0x7f2bb0075f70 2024-04-26T20:33:34.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.164+0000 7f2ba6a00700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2b940a3d50 0x7f2b940a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.164+0000 7f2ba6a00700 1 -- 172.21.15.102:0/1996229943 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2b9c0d08c0 con 0x7f2bb0075f70 2024-04-26T20:33:34.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.164+0000 7f2bade00700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2b940a3d50 0x7f2b940a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.164+0000 7f2ba6a00700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] conn(0x7f2b940aeeb0 0x7f2b940b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.164+0000 7f2ba6a00700 1 -- 172.21.15.102:0/1996229943 --> [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b9c022e50 con 0x7f2b940aeeb0 2024-04-26T20:33:34.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.164+0000 7f2ba6a00700 1 -- 172.21.15.102:0/1996229943 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f2b9c0d52c0 con 0x7f2bb0075f70 2024-04-26T20:33:34.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.165+0000 7f2bae800700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] conn(0x7f2b940aeeb0 0x7f2b940b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.165+0000 7f2bade00700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2b940a3d50 0x7f2b940a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009d60 tx=0x7f2ba0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:34.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.165+0000 7f2bae800700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] conn(0x7f2b940aeeb0 0x7f2b940b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:34.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.166+0000 7f2ba6a00700 1 -- 172.21.15.102:0/1996229943 <== osd.7 v2:172.21.15.112:6816/971991599 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2b9c022e50 con 0x7f2b940aeeb0 2024-04-26T20:33:34.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.188+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 --> [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2b90002e60 con 0x7f2b940aeeb0 2024-04-26T20:33:34.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.189+0000 7f2ba6a00700 1 -- 172.21.15.102:0/1996229943 <== osd.7 v2:172.21.15.112:6816/971991599 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2b90002e60 con 0x7f2b940aeeb0 2024-04-26T20:33:34.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 >> [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] conn(0x7f2b940aeeb0 msgr2=0x7f2b940b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bb5000700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] conn(0x7f2b940aeeb0 0x7f2b940b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2b940a3d50 msgr2=0x7f2b940a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bb5000700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2b940a3d50 0x7f2b940a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009d60 tx=0x7f2ba0009260 comp rx=0 tx=0).stop 2024-04-26T20:33:34.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb0075f70 msgr2=0x7f2bb01a0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bb5000700 1 --2- 172.21.15.102:0/1996229943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb0075f70 0x7f2bb01a0600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c00e9f0 tx=0x7f2b9c00ed00 comp rx=0 tx=0).stop 2024-04-26T20:33:34.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bae800700 1 -- 172.21.15.102:0/1996229943 reap_dead start 2024-04-26T20:33:34.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 shutdown_connections 2024-04-26T20:33:34.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 >> 172.21.15.102:0/1996229943 conn(0x7f2bb007a580 msgr2=0x7f2bb0110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 shutdown_connections 2024-04-26T20:33:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.190+0000 7f2bb5000700 1 -- 172.21.15.102:0/1996229943 wait complete. 2024-04-26T20:33:34.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.728+0000 7feee8600700 1 -- 172.21.15.102:0/2045161187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee0103e30 msgr2=0x7feee0110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.728+0000 7feee8600700 1 --2- 172.21.15.102:0/2045161187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee0103e30 0x7feee0110290 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7feed8009f90 tx=0x7feed800b330 comp rx=0 tx=0).stop 2024-04-26T20:33:34.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.728+0000 7feee8600700 1 -- 172.21.15.102:0/2045161187 shutdown_connections 2024-04-26T20:33:34.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.728+0000 7feee8600700 1 --2- 172.21.15.102:0/2045161187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee0103e30 0x7feee0110290 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.728+0000 7feee8600700 1 --2- 172.21.15.102:0/2045161187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feee01094c0 0x7feee01098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.728+0000 7feee8600700 1 --2- 172.21.15.102:0/2045161187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feee0103470 0x7feee01038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.728+0000 7feee8600700 1 -- 172.21.15.102:0/2045161187 >> 172.21.15.102:0/2045161187 conn(0x7feee007a580 msgr2=0x7feee007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:34.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.728+0000 7feee8600700 1 -- 172.21.15.102:0/2045161187 shutdown_connections 2024-04-26T20:33:34.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.728+0000 7feee8600700 1 -- 172.21.15.102:0/2045161187 wait complete. 2024-04-26T20:33:34.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.729+0000 7feee8600700 1 Processor -- start 2024-04-26T20:33:34.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee8600700 1 -- start start 2024-04-26T20:33:34.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee8600700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feee0103470 0x7feee01a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee8600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feee0103e30 0x7feee01a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee8600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee01094c0 0x7feee019be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee8600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feee0075f50 con 0x7feee01094c0 2024-04-26T20:33:34.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee8600700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feee0075dd0 con 0x7feee0103470 2024-04-26T20:33:34.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee8600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7feee00760d0 con 0x7feee0103e30 2024-04-26T20:33:34.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee5c00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feee0103470 0x7feee01a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee6600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee01094c0 0x7feee019be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee5c00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feee0103470 0x7feee01a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:39792/0 (socket says 172.21.15.102:39792) 2024-04-26T20:33:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee5c00700 1 -- 172.21.15.102:0/2384307436 learned_addr learned my addr 172.21.15.102:0/2384307436 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.730+0000 7feee5200700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feee0103e30 0x7feee01a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.731+0000 7feee5c00700 1 -- 172.21.15.102:0/2384307436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feee0103e30 msgr2=0x7feee01a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.731+0000 7feee5c00700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feee0103e30 0x7feee01a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.731+0000 7feee5c00700 1 -- 172.21.15.102:0/2384307436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee01094c0 msgr2=0x7feee019be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.731+0000 7feee5c00700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee01094c0 0x7feee019be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.731+0000 7feee5c00700 1 -- 172.21.15.102:0/2384307436 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee019c5d0 con 0x7feee0103470 2024-04-26T20:33:34.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.731+0000 7feee5200700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feee0103e30 0x7feee01a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:34.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.731+0000 7feee5c00700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feee0103470 0x7feee01a0890 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feed400e9c0 tx=0x7feed400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:34.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.731+0000 7feede000700 1 -- 172.21.15.102:0/2384307436 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4004a30 con 0x7feee0103470 2024-04-26T20:33:34.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.731+0000 7feede000700 1 -- 172.21.15.102:0/2384307436 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feed400cc70 con 0x7feee0103470 2024-04-26T20:33:34.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.731+0000 7feede000700 1 -- 172.21.15.102:0/2384307436 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4005490 con 0x7feee0103470 2024-04-26T20:33:34.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.732+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee019c8c0 con 0x7feee0103470 2024-04-26T20:33:34.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.732+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee01a8810 con 0x7feee0103470 2024-04-26T20:33:34.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.732+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feee0108320 con 0x7feee0103470 2024-04-26T20:33:34.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.735+0000 7feede000700 1 -- 172.21.15.102:0/2384307436 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7feed4004bd0 con 0x7feee0103470 2024-04-26T20:33:34.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.736+0000 7feede000700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7feec80a3d50 0x7feec80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.736+0000 7feede000700 1 -- 172.21.15.102:0/2384307436 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7feed40d0d30 con 0x7feee0103470 2024-04-26T20:33:34.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.736+0000 7feee5200700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7feec80a3d50 0x7feec80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.737+0000 7feede000700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] conn(0x7feec80aeea0 0x7feec80b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:34.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.737+0000 7feede000700 1 -- 172.21.15.102:0/2384307436 --> [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feed401ee50 con 0x7feec80aeea0 2024-04-26T20:33:34.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.737+0000 7feede000700 1 -- 172.21.15.102:0/2384307436 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7feed40d10e0 con 0x7feee0103470 2024-04-26T20:33:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.737+0000 7feee6600700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] conn(0x7feec80aeea0 0x7feec80b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.737+0000 7feee5200700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7feec80a3d50 0x7feec80a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feecc009680 tx=0x7feecc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.738+0000 7feee6600700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] conn(0x7feec80aeea0 0x7feec80b1340 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:34.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.739+0000 7feede000700 1 -- 172.21.15.102:0/2384307436 <== osd.10 v2:172.21.15.112:6824/2204137717 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feed401ee50 con 0x7feec80aeea0 2024-04-26T20:33:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.757+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 --> [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feee0104b60 con 0x7feec80aeea0 2024-04-26T20:33:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.757+0000 7feede000700 1 -- 172.21.15.102:0/2384307436 <== osd.10 v2:172.21.15.112:6824/2204137717 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7feee0104b60 con 0x7feec80aeea0 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 >> [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] conn(0x7feec80aeea0 msgr2=0x7feec80b1340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee8600700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] conn(0x7feec80aeea0 0x7feec80b1340 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7feec80a3d50 msgr2=0x7feec80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee8600700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7feec80a3d50 0x7feec80a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feecc009680 tx=0x7feecc009260 comp rx=0 tx=0).stop 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feee0103470 msgr2=0x7feee01a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee8600700 1 --2- 172.21.15.102:0/2384307436 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feee0103470 0x7feee01a0890 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feed400e9c0 tx=0x7feed400ecd0 comp rx=0 tx=0).stop 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee6600700 1 -- 172.21.15.102:0/2384307436 reap_dead start 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 shutdown_connections 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 >> 172.21.15.102:0/2384307436 conn(0x7feee007a580 msgr2=0x7feee0101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 shutdown_connections 2024-04-26T20:33:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:34.758+0000 7feee8600700 1 -- 172.21.15.102:0/2384307436 wait complete. 2024-04-26T20:33:35.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.435+0000 7fb31ca00700 1 -- 172.21.15.102:0/1614598433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb318109490 msgr2=0x7fb318109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:35.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.435+0000 7fb31ca00700 1 --2- 172.21.15.102:0/1614598433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb318109490 0x7fb318109870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb308009990 tx=0x7fb308009ca0 comp rx=0 tx=0).stop 2024-04-26T20:33:35.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.435+0000 7fb31ca00700 1 -- 172.21.15.102:0/1614598433 shutdown_connections 2024-04-26T20:33:35.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.435+0000 7fb31ca00700 1 --2- 172.21.15.102:0/1614598433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb31806c360 0x7fb3181102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.435+0000 7fb31ca00700 1 --2- 172.21.15.102:0/1614598433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb318109490 0x7fb318109870 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.435+0000 7fb31ca00700 1 --2- 172.21.15.102:0/1614598433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31806b9c0 0x7fb31806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.435+0000 7fb31ca00700 1 -- 172.21.15.102:0/1614598433 >> 172.21.15.102:0/1614598433 conn(0x7fb318079290 msgr2=0x7fb3180796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.435+0000 7fb31ca00700 1 -- 172.21.15.102:0/1614598433 shutdown_connections 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.435+0000 7fb31ca00700 1 -- 172.21.15.102:0/1614598433 wait complete. 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.437+0000 7fb31ca00700 1 Processor -- start 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.437+0000 7fb31ca00700 1 -- start start 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.437+0000 7fb31ca00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb31806b9c0 0x7fb3181023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.437+0000 7fb31ca00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31806c360 0x7fb318102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.437+0000 7fb31ca00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb318109490 0x7fb318102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.437+0000 7fb31ca00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181135a0 con 0x7fb31806b9c0 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.437+0000 7fb31ca00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb318113420 con 0x7fb318109490 2024-04-26T20:33:35.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.437+0000 7fb31ca00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb318113720 con 0x7fb31806c360 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb315e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb31806b9c0 0x7fb3181023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb315e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb31806b9c0 0x7fb3181023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58634/0 (socket says 172.21.15.102:58634) 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb315e00700 1 -- 172.21.15.102:0/1441713421 learned_addr learned my addr 172.21.15.102:0/1441713421 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb315400700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31806c360 0x7fb318102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb316800700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb318109490 0x7fb318102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb315e00700 1 -- 172.21.15.102:0/1441713421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31806c360 msgr2=0x7fb318102920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb315e00700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31806c360 0x7fb318102920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb315e00700 1 -- 172.21.15.102:0/1441713421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb318109490 msgr2=0x7fb318102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb315e00700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb318109490 0x7fb318102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb315e00700 1 -- 172.21.15.102:0/1441713421 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3181a8020 con 0x7fb31806b9c0 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb316800700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb318109490 0x7fb318102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.438+0000 7fb315400700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31806c360 0x7fb318102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:35.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.439+0000 7fb315e00700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb31806b9c0 0x7fb3181023e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb3080057f0 tx=0x7fb30800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.439+0000 7fb30ea00700 1 -- 172.21.15.102:0/1441713421 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308026070 con 0x7fb31806b9c0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.439+0000 7fb30ea00700 1 -- 172.21.15.102:0/1441713421 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb308021410 con 0x7fb31806b9c0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.439+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3181a82b0 con 0x7fb31806b9c0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.439+0000 7fb30ea00700 1 -- 172.21.15.102:0/1441713421 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308017650 con 0x7fb31806b9c0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.439+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3181a87e0 con 0x7fb31806b9c0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.440+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb318063b30 con 0x7fb31806b9c0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.442+0000 7fb30ea00700 1 -- 172.21.15.102:0/1441713421 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fb308029070 con 0x7fb31806b9c0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.444+0000 7fb30ea00700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb2fc0a3d50 0x7fb2fc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.444+0000 7fb30ea00700 1 -- 172.21.15.102:0/1441713421 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb3080d1a40 con 0x7fb31806b9c0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.444+0000 7fb315400700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb2fc0a3d50 0x7fb2fc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.444+0000 7fb30ea00700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] conn(0x7fb2fc0aed80 0x7fb2fc0b1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.444+0000 7fb30ea00700 1 -- 172.21.15.102:0/1441713421 --> [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2fc0b18f0 con 0x7fb2fc0aed80 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.444+0000 7fb30ea00700 1 -- 172.21.15.102:0/1441713421 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb3080d1df0 con 0x7fb31806b9c0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.444+0000 7fb316800700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] conn(0x7fb2fc0aed80 0x7fb2fc0b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:35.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.445+0000 7fb315400700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb2fc0a3d50 0x7fb2fc0a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb318079970 tx=0x7fb304009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:35.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.445+0000 7fb316800700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] conn(0x7fb2fc0aed80 0x7fb2fc0b1220 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:35.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.446+0000 7fb30ea00700 1 -- 172.21.15.102:0/1441713421 <== osd.9 v2:172.21.15.102:6826/4267695892 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb2fc0b18f0 con 0x7fb2fc0aed80 2024-04-26T20:33:35.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.464+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 --> [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb318063f30 con 0x7fb2fc0aed80 2024-04-26T20:33:35.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.465+0000 7fb30ea00700 1 -- 172.21.15.102:0/1441713421 <== osd.9 v2:172.21.15.102:6826/4267695892 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb318063f30 con 0x7fb2fc0aed80 2024-04-26T20:33:35.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] conn(0x7fb2fc0aed80 msgr2=0x7fb2fc0b1220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:35.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb31ca00700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] conn(0x7fb2fc0aed80 0x7fb2fc0b1220 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:35.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb2fc0a3d50 msgr2=0x7fb2fc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:35.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb31ca00700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb2fc0a3d50 0x7fb2fc0a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb318079970 tx=0x7fb304009260 comp rx=0 tx=0).stop 2024-04-26T20:33:35.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb31806b9c0 msgr2=0x7fb3181023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:35.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb31ca00700 1 --2- 172.21.15.102:0/1441713421 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb31806b9c0 0x7fb3181023e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb3080057f0 tx=0x7fb30800bfd0 comp rx=0 tx=0).stop 2024-04-26T20:33:35.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb316800700 1 -- 172.21.15.102:0/1441713421 reap_dead start 2024-04-26T20:33:35.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 shutdown_connections 2024-04-26T20:33:35.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 >> 172.21.15.102:0/1441713421 conn(0x7fb318079290 msgr2=0x7fb3181004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:35.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 shutdown_connections 2024-04-26T20:33:35.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:35.466+0000 7fb31ca00700 1 -- 172.21.15.102:0/1441713421 wait complete. 2024-04-26T20:33:35.974 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:35 smithi102 ceph-mon[28094]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 77 B/s, 0 objects/s recovering 2024-04-26T20:33:36.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:35 smithi112 ceph-mon[32769]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 77 B/s, 0 objects/s recovering 2024-04-26T20:33:36.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:35 smithi130 ceph-mon[32704]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 77 B/s, 0 objects/s recovering 2024-04-26T20:33:36.597 INFO:teuthology.orchestra.run.smithi102.stdout:343597383709 2024-04-26T20:33:36.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T20:33:36.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:36 smithi102 ceph-mon[28094]: pgmap v355: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 77 B/s, 0 objects/s recovering 2024-04-26T20:33:37.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:36 smithi102 ceph-mon[28094]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-26T20:33:37.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:36 smithi102 ceph-mon[28094]: Cluster is now healthy 2024-04-26T20:33:37.291 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:36 smithi130 ceph-mon[32704]: pgmap v355: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 77 B/s, 0 objects/s recovering 2024-04-26T20:33:37.291 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:36 smithi130 ceph-mon[32704]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-26T20:33:37.291 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:36 smithi130 ceph-mon[32704]: Cluster is now healthy 2024-04-26T20:33:37.291 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:36 smithi112 ceph-mon[32769]: pgmap v355: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 77 B/s, 0 objects/s recovering 2024-04-26T20:33:37.291 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:36 smithi112 ceph-mon[32769]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-26T20:33:37.291 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:36 smithi112 ceph-mon[32769]: Cluster is now healthy 2024-04-26T20:33:37.941 INFO:teuthology.orchestra.run.smithi102.stdout:163208757309 2024-04-26T20:33:37.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T20:33:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.517+0000 7fa128400700 1 -- 172.21.15.102:0/621242534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1201094c0 msgr2=0x7fa1201098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.517+0000 7fa128400700 1 --2- 172.21.15.102:0/621242534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1201094c0 0x7fa1201098a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa11400b0a0 tx=0x7fa11400b3b0 comp rx=0 tx=0).stop 2024-04-26T20:33:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.517+0000 7fa128400700 1 -- 172.21.15.102:0/621242534 shutdown_connections 2024-04-26T20:33:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.517+0000 7fa128400700 1 --2- 172.21.15.102:0/621242534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa120103e30 0x7fa120110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.517+0000 7fa128400700 1 --2- 172.21.15.102:0/621242534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1201094c0 0x7fa1201098a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.517+0000 7fa128400700 1 --2- 172.21.15.102:0/621242534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa120103470 0x7fa1201038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.517+0000 7fa128400700 1 -- 172.21.15.102:0/621242534 >> 172.21.15.102:0/621242534 conn(0x7fa12007a580 msgr2=0x7fa12007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.517+0000 7fa128400700 1 -- 172.21.15.102:0/621242534 shutdown_connections 2024-04-26T20:33:39.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.517+0000 7fa128400700 1 -- 172.21.15.102:0/621242534 wait complete. 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.518+0000 7fa128400700 1 Processor -- start 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.518+0000 7fa128400700 1 -- start start 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa128400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa120103470 0x7fa120075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa128400700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa120103e30 0x7fa120075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa128400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1201094c0 0x7fa1200761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa128400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa120113610 con 0x7fa1201094c0 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa128400700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa120113490 con 0x7fa120103e30 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa128400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa120113790 con 0x7fa120103470 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa126400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1201094c0 0x7fa1200761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa126400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1201094c0 0x7fa1200761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58652/0 (socket says 172.21.15.102:58652) 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa125a00700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa120103470 0x7fa120075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa126400700 1 -- 172.21.15.102:0/1733471291 learned_addr learned my addr 172.21.15.102:0/1733471291 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.519+0000 7fa125000700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa120103e30 0x7fa120075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.520+0000 7fa125000700 1 -- 172.21.15.102:0/1733471291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa120103470 msgr2=0x7fa120075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.520+0000 7fa125000700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa120103470 0x7fa120075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.520+0000 7fa125000700 1 -- 172.21.15.102:0/1733471291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1201094c0 msgr2=0x7fa1200761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.520+0000 7fa125000700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1201094c0 0x7fa1200761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.520+0000 7fa125000700 1 -- 172.21.15.102:0/1733471291 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1201a8160 con 0x7fa120103e30 2024-04-26T20:33:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.520+0000 7fa126400700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1201094c0 0x7fa1200761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:33:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.520+0000 7fa125000700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa120103e30 0x7fa120075c60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00e9f0 tx=0x7fa10c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.520+0000 7fa11e000700 1 -- 172.21.15.102:0/1733471291 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c00cb70 con 0x7fa120103e30 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.520+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1201a8450 con 0x7fa120103e30 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.520+0000 7fa11e000700 1 -- 172.21.15.102:0/1733471291 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa10c00ee90 con 0x7fa120103e30 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.521+0000 7fa11e000700 1 -- 172.21.15.102:0/1733471291 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c003d40 con 0x7fa120103e30 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.521+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1201a8990 con 0x7fa120103e30 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.522+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa120063b30 con 0x7fa120103e30 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.524+0000 7fa11e000700 1 -- 172.21.15.102:0/1733471291 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa10c003ee0 con 0x7fa120103e30 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.525+0000 7fa11e000700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1080a3cd0 0x7fa1080a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.525+0000 7fa11e000700 1 -- 172.21.15.102:0/1733471291 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa10c0d0d70 con 0x7fa120103e30 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.525+0000 7fa125a00700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1080a3cd0 0x7fa1080a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.526+0000 7fa11e000700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] conn(0x7fa1080aee30 0x7fa1080b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.526+0000 7fa11e000700 1 -- 172.21.15.102:0/1733471291 --> [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa1080b19a0 con 0x7fa1080aee30 2024-04-26T20:33:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.526+0000 7fa11e000700 1 -- 172.21.15.102:0/1733471291 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa10c0d1120 con 0x7fa120103e30 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.526+0000 7fa125a00700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1080a3cd0 0x7fa1080a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa1140062a0 tx=0x7fa114006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.526+0000 7fa126400700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] conn(0x7fa1080aee30 0x7fa1080b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.527+0000 7fa126400700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] conn(0x7fa1080aee30 0x7fa1080b12d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.528+0000 7fa11e000700 1 -- 172.21.15.102:0/1733471291 <== osd.2 v2:172.21.15.130:6800/2227083221 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa1080b19a0 con 0x7fa1080aee30 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.546+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 --> [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa120063f30 con 0x7fa1080aee30 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.547+0000 7fa11e000700 1 -- 172.21.15.102:0/1733471291 <== osd.2 v2:172.21.15.130:6800/2227083221 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa120063f30 con 0x7fa1080aee30 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.547+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 >> [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] conn(0x7fa1080aee30 msgr2=0x7fa1080b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.547+0000 7fa128400700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] conn(0x7fa1080aee30 0x7fa1080b12d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.547+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1080a3cd0 msgr2=0x7fa1080a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.547+0000 7fa128400700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa1080a3cd0 0x7fa1080a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa1140062a0 tx=0x7fa114006210 comp rx=0 tx=0).stop 2024-04-26T20:33:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.547+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa120103e30 msgr2=0x7fa120075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:39.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.547+0000 7fa128400700 1 --2- 172.21.15.102:0/1733471291 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa120103e30 0x7fa120075c60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00e9f0 tx=0x7fa10c00ed00 comp rx=0 tx=0).stop 2024-04-26T20:33:39.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.547+0000 7fa126400700 1 -- 172.21.15.102:0/1733471291 reap_dead start 2024-04-26T20:33:39.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.548+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 shutdown_connections 2024-04-26T20:33:39.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.548+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 >> 172.21.15.102:0/1733471291 conn(0x7fa12007a580 msgr2=0x7fa120102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:39.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.548+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 shutdown_connections 2024-04-26T20:33:39.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:37.548+0000 7fa128400700 1 -- 172.21.15.102:0/1733471291 wait complete. 2024-04-26T20:33:39.386 INFO:teuthology.orchestra.run.smithi102.stdout:309237645345 2024-04-26T20:33:39.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T20:33:39.860 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:39 smithi102 ceph-mon[28094]: pgmap v356: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 67 B/s, 0 objects/s recovering 2024-04-26T20:33:39.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:39 smithi130 ceph-mon[32704]: pgmap v356: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 67 B/s, 0 objects/s recovering 2024-04-26T20:33:40.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:39 smithi112 ceph-mon[32769]: pgmap v356: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 67 B/s, 0 objects/s recovering 2024-04-26T20:33:40.028 INFO:teuthology.orchestra.run.smithi102.stdout:287762808871 2024-04-26T20:33:40.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T20:33:40.378 INFO:teuthology.orchestra.run.smithi102.stdout:219043332146 2024-04-26T20:33:40.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T20:33:41.828 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:41 smithi102 ceph-mon[28094]: pgmap v357: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 61 B/s, 0 objects/s recovering 2024-04-26T20:33:41.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:41 smithi130 ceph-mon[32704]: pgmap v357: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 61 B/s, 0 objects/s recovering 2024-04-26T20:33:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:41 smithi112 ceph-mon[32769]: pgmap v357: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 61 B/s, 0 objects/s recovering 2024-04-26T20:33:42.330 INFO:teuthology.orchestra.run.smithi102.stdout:253403070509 2024-04-26T20:33:42.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T20:33:42.828 INFO:teuthology.orchestra.run.smithi102.stdout:77309411409 2024-04-26T20:33:42.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T20:33:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.366+0000 7f841c400700 1 -- 172.21.15.102:0/1104405825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141094f0 msgr2=0x7f84141098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.366+0000 7f841c400700 1 --2- 172.21.15.102:0/1104405825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141094f0 0x7f84141098d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8408009990 tx=0x7f8408009ca0 comp rx=0 tx=0).stop 2024-04-26T20:33:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.366+0000 7f841c400700 1 -- 172.21.15.102:0/1104405825 shutdown_connections 2024-04-26T20:33:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.366+0000 7f841c400700 1 --2- 172.21.15.102:0/1104405825 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414103e30 0x7f84141102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.366+0000 7f841c400700 1 --2- 172.21.15.102:0/1104405825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141094f0 0x7f84141098d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.366+0000 7f841c400700 1 --2- 172.21.15.102:0/1104405825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8414103470 0x7f84141038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.366+0000 7f841c400700 1 -- 172.21.15.102:0/1104405825 >> 172.21.15.102:0/1104405825 conn(0x7f841407a5d0 msgr2=0x7f841407a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:43.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.366+0000 7f841c400700 1 -- 172.21.15.102:0/1104405825 shutdown_connections 2024-04-26T20:33:43.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.367+0000 7f841c400700 1 -- 172.21.15.102:0/1104405825 wait complete. 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f841c400700 1 Processor -- start 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f841c400700 1 -- start start 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f841c400700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8414103470 0x7f84141a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f841c400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414103e30 0x7f84141a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f841c400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141094f0 0x7f841410f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f841c400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8414113640 con 0x7f84141094f0 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f841c400700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f84141134c0 con 0x7f8414103470 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f841c400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f84141137c0 con 0x7f8414103e30 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f8419a00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8414103470 0x7f84141a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f8419a00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8414103470 0x7f84141a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:45462/0 (socket says 172.21.15.102:45462) 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.368+0000 7f8419a00700 1 -- 172.21.15.102:0/4174620661 learned_addr learned my addr 172.21.15.102:0/4174620661 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.369+0000 7f841a400700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141094f0 0x7f841410f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.369+0000 7f8419000700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414103e30 0x7f84141a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:43.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.369+0000 7f8419a00700 1 -- 172.21.15.102:0/4174620661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414103e30 msgr2=0x7f84141a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.369+0000 7f8419a00700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414103e30 0x7f84141a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.369+0000 7f8419a00700 1 -- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141094f0 msgr2=0x7f841410f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.369+0000 7f8419a00700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141094f0 0x7f841410f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.369+0000 7f8419a00700 1 -- 172.21.15.102:0/4174620661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841410ff90 con 0x7f8414103470 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.369+0000 7f8419a00700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8414103470 0x7f84141a0830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8408005910 tx=0x7f840800b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.370+0000 7f8412000700 1 -- 172.21.15.102:0/4174620661 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8408027070 con 0x7f8414103470 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.370+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8414110220 con 0x7f8414103470 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.370+0000 7f8412000700 1 -- 172.21.15.102:0/4174620661 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8408022470 con 0x7f8414103470 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.370+0000 7f8412000700 1 -- 172.21.15.102:0/4174620661 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8408017650 con 0x7f8414103470 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.370+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f841419c390 con 0x7f8414103470 2024-04-26T20:33:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.371+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8414102990 con 0x7f8414103470 2024-04-26T20:33:43.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.373+0000 7f8412000700 1 -- 172.21.15.102:0/4174620661 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f840802a070 con 0x7f8414103470 2024-04-26T20:33:43.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.374+0000 7f8412000700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3cd0 0x7f83fc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:43.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.374+0000 7f8412000700 1 -- 172.21.15.102:0/4174620661 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f84080d1b60 con 0x7f8414103470 2024-04-26T20:33:43.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.375+0000 7f8412000700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] conn(0x7f83fc0aee30 0x7f83fc0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:43.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.375+0000 7f8412000700 1 -- 172.21.15.102:0/4174620661 --> [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8408024ec0 con 0x7f83fc0aee30 2024-04-26T20:33:43.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.375+0000 7f8412000700 1 -- 172.21.15.102:0/4174620661 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f84080d1f10 con 0x7f8414103470 2024-04-26T20:33:43.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.375+0000 7f8419000700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3cd0 0x7f83fc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.375+0000 7f841a400700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] conn(0x7f83fc0aee30 0x7f83fc0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.376+0000 7f841a400700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] conn(0x7f83fc0aee30 0x7f83fc0b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.376+0000 7f8419000700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3cd0 0x7f83fc0a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8414110630 tx=0x7f8400009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.377+0000 7f8412000700 1 -- 172.21.15.102:0/4174620661 <== osd.3 v2:172.21.15.102:6810/819862253 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8408024ec0 con 0x7f83fc0aee30 2024-04-26T20:33:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.395+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 --> [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8414104b90 con 0x7f83fc0aee30 2024-04-26T20:33:43.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.396+0000 7f8412000700 1 -- 172.21.15.102:0/4174620661 <== osd.3 v2:172.21.15.102:6810/819862253 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8414104b90 con 0x7f83fc0aee30 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.396+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] conn(0x7f83fc0aee30 msgr2=0x7f83fc0b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.396+0000 7f841c400700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] conn(0x7f83fc0aee30 0x7f83fc0b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.396+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3cd0 msgr2=0x7f83fc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.396+0000 7f841c400700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3cd0 0x7f83fc0a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8414110630 tx=0x7f8400009260 comp rx=0 tx=0).stop 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.396+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8414103470 msgr2=0x7f84141a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.396+0000 7f841c400700 1 --2- 172.21.15.102:0/4174620661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8414103470 0x7f84141a0830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8408005910 tx=0x7f840800b670 comp rx=0 tx=0).stop 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.396+0000 7f841a400700 1 -- 172.21.15.102:0/4174620661 reap_dead start 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.396+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 shutdown_connections 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.396+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 >> 172.21.15.102:0/4174620661 conn(0x7f841407a5d0 msgr2=0x7f8414101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.397+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 shutdown_connections 2024-04-26T20:33:43.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:43.397+0000 7f841c400700 1 -- 172.21.15.102:0/4174620661 wait complete. 2024-04-26T20:33:43.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:43 smithi130 ceph-mon[32704]: pgmap v358: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 51 B/s, 0 objects/s recovering 2024-04-26T20:33:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:43 smithi102 ceph-mon[28094]: pgmap v358: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 51 B/s, 0 objects/s recovering 2024-04-26T20:33:44.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:43 smithi112 ceph-mon[32769]: pgmap v358: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 51 B/s, 0 objects/s recovering 2024-04-26T20:33:44.578 INFO:teuthology.orchestra.run.smithi102.stdout:107374182475 2024-04-26T20:33:44.579 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T20:33:44.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.159+0000 7f88c0c00700 1 -- 172.21.15.102:0/4122631805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88bc10fb80 msgr2=0x7f88bc111ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:44.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.159+0000 7f88c0c00700 1 --2- 172.21.15.102:0/4122631805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88bc10fb80 0x7f88bc111ff0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f88b40098f0 tx=0x7f88b4009c00 comp rx=0 tx=0).stop 2024-04-26T20:33:44.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.160+0000 7f88c0c00700 1 -- 172.21.15.102:0/4122631805 shutdown_connections 2024-04-26T20:33:44.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.160+0000 7f88c0c00700 1 --2- 172.21.15.102:0/4122631805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88bc10fb80 0x7f88bc111ff0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.160+0000 7f88c0c00700 1 --2- 172.21.15.102:0/4122631805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88bc075850 0x7f88bc075c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.160+0000 7f88c0c00700 1 --2- 172.21.15.102:0/4122631805 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc076170 0x7f88bc10f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.160+0000 7f88c0c00700 1 -- 172.21.15.102:0/4122631805 >> 172.21.15.102:0/4122631805 conn(0x7f88bc0fee70 msgr2=0x7f88bc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.160+0000 7f88c0c00700 1 -- 172.21.15.102:0/4122631805 shutdown_connections 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.160+0000 7f88c0c00700 1 -- 172.21.15.102:0/4122631805 wait complete. 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.161+0000 7f88c0c00700 1 Processor -- start 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.161+0000 7f88c0c00700 1 -- start start 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.162+0000 7f88c0c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88bc075850 0x7f88bc1a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.162+0000 7f88c0c00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc076170 0x7f88bc1a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.162+0000 7f88c0c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88bc10fb80 0x7f88bc19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.162+0000 7f88c0c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc115790 con 0x7f88bc10fb80 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.162+0000 7f88c0c00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc115610 con 0x7f88bc076170 2024-04-26T20:33:44.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.162+0000 7f88c0c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc115910 con 0x7f88bc075850 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.162+0000 7f88bbe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88bc10fb80 0x7f88bc19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.162+0000 7f88baa00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc076170 0x7f88bc1a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.162+0000 7f88baa00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc076170 0x7f88bc1a1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:45474/0 (socket says 172.21.15.102:45474) 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.162+0000 7f88baa00700 1 -- 172.21.15.102:0/1871920145 learned_addr learned my addr 172.21.15.102:0/1871920145 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88baa00700 1 -- 172.21.15.102:0/1871920145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88bc075850 msgr2=0x7f88bc1a1930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88baa00700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88bc075850 0x7f88bc1a1930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88baa00700 1 -- 172.21.15.102:0/1871920145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88bc10fb80 msgr2=0x7f88bc19baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88baa00700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88bc10fb80 0x7f88bc19baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88baa00700 1 -- 172.21.15.102:0/1871920145 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88bc19c250 con 0x7f88bc076170 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88baa00700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc076170 0x7f88bc1a1e70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f88a800d980 tx=0x7f88a800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88b3e00700 1 -- 172.21.15.102:0/1871920145 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8016070 con 0x7f88bc076170 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88b3e00700 1 -- 172.21.15.102:0/1871920145 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a8005020 con 0x7f88bc076170 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88bc19c540 con 0x7f88bc076170 2024-04-26T20:33:44.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88b3e00700 1 -- 172.21.15.102:0/1871920145 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8009ab0 con 0x7f88bc076170 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.163+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88bc1a8800 con 0x7f88bc076170 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.164+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f88bc063b70 con 0x7f88bc076170 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.167+0000 7f88b3e00700 1 -- 172.21.15.102:0/1871920145 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f88a8009c50 con 0x7f88bc076170 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.168+0000 7f88b3e00700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88a00a3cd0 0x7f88a00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.169+0000 7f88b3e00700 1 -- 172.21.15.102:0/1871920145 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f88a80d0900 con 0x7f88bc076170 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.169+0000 7f88bb400700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88a00a3cd0 0x7f88a00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.169+0000 7f88b3e00700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] conn(0x7f88a00aee30 0x7f88a00b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.169+0000 7f88b3e00700 1 -- 172.21.15.102:0/1871920145 --> [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88a8014e80 con 0x7f88a00aee30 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.169+0000 7f88b3e00700 1 -- 172.21.15.102:0/1871920145 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f88a80d0cb0 con 0x7f88bc076170 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.170+0000 7f88bbe00700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] conn(0x7f88a00aee30 0x7f88a00b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:44.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.170+0000 7f88bb400700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88a00a3cd0 0x7f88a00a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f88ac00e350 tx=0x7f88ac00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.170+0000 7f88bbe00700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] conn(0x7f88a00aee30 0x7f88a00b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.171+0000 7f88b3e00700 1 -- 172.21.15.102:0/1871920145 <== osd.1 v2:172.21.15.112:6800/3112086050 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f88a8014e80 con 0x7f88a00aee30 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.190+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 --> [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f88bc063ff0 con 0x7f88a00aee30 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.190+0000 7f88b3e00700 1 -- 172.21.15.102:0/1871920145 <== osd.1 v2:172.21.15.112:6800/3112086050 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f88bc063ff0 con 0x7f88a00aee30 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.191+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 >> [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] conn(0x7f88a00aee30 msgr2=0x7f88a00b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.191+0000 7f88c0c00700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] conn(0x7f88a00aee30 0x7f88a00b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.191+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88a00a3cd0 msgr2=0x7f88a00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.191+0000 7f88c0c00700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88a00a3cd0 0x7f88a00a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f88ac00e350 tx=0x7f88ac00b040 comp rx=0 tx=0).stop 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.191+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc076170 msgr2=0x7f88bc1a1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.191+0000 7f88c0c00700 1 --2- 172.21.15.102:0/1871920145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88bc076170 0x7f88bc1a1e70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f88a800d980 tx=0x7f88a800dc90 comp rx=0 tx=0).stop 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.191+0000 7f88bbe00700 1 -- 172.21.15.102:0/1871920145 reap_dead start 2024-04-26T20:33:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.191+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 shutdown_connections 2024-04-26T20:33:44.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.191+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 >> 172.21.15.102:0/1871920145 conn(0x7f88bc0fee70 msgr2=0x7f88bc0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:44.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.192+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 shutdown_connections 2024-04-26T20:33:44.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:44.192+0000 7f88c0c00700 1 -- 172.21.15.102:0/1871920145 wait complete. 2024-04-26T20:33:45.147 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:45.149 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:45.519 INFO:teuthology.orchestra.run.smithi102.stdout:55834574934 2024-04-26T20:33:45.519 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T20:33:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:45.726 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:45.734 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:45 smithi102 ceph-mon[28094]: pgmap v359: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:33:45.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:45 smithi130 ceph-mon[32704]: pgmap v359: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:33:46.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:45 smithi112 ceph-mon[32769]: pgmap v359: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:33:46.642 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:46.648 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:46.651 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:46.662 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:47.739 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:47 smithi102 ceph-mon[28094]: pgmap v360: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:33:47.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:47 smithi130 ceph-mon[32704]: pgmap v360: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:33:48.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:47 smithi112 ceph-mon[32769]: pgmap v360: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:33:49.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:49 smithi102 ceph-mon[28094]: pgmap v361: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:33:50.022 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:49 smithi112 ceph-mon[32769]: pgmap v361: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:33:50.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:49 smithi130 ceph-mon[32704]: pgmap v361: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:33:51.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:51 smithi102 ceph-mon[28094]: pgmap v362: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:33:52.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:51 smithi112 ceph-mon[32769]: pgmap v362: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:33:52.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:51 smithi130 ceph-mon[32704]: pgmap v362: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:33:52.853 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:33:53.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:53 smithi102 ceph-mon[28094]: pgmap v363: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:33:54.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:53 smithi112 ceph-mon[32769]: pgmap v363: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:33:54.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:53 smithi130 ceph-mon[32704]: pgmap v363: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:33:55.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.684+0000 7fadc1e00700 1 -- 172.21.15.102:0/2166018569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadbc10fe50 msgr2=0x7fadbc112240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:55.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.684+0000 7fadc1e00700 1 --2- 172.21.15.102:0/2166018569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadbc10fe50 0x7fadbc112240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fadb400b0a0 tx=0x7fadb400b3b0 comp rx=0 tx=0).stop 2024-04-26T20:33:55.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.685+0000 7fadc1e00700 1 -- 172.21.15.102:0/2166018569 shutdown_connections 2024-04-26T20:33:55.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.685+0000 7fadc1e00700 1 --2- 172.21.15.102:0/2166018569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadbc10fe50 0x7fadbc112240 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.685+0000 7fadc1e00700 1 --2- 172.21.15.102:0/2166018569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadbc075650 0x7fadbc075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.685+0000 7fadc1e00700 1 --2- 172.21.15.102:0/2166018569 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fadbc075f70 0x7fadbc10f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.685+0000 7fadc1e00700 1 -- 172.21.15.102:0/2166018569 >> 172.21.15.102:0/2166018569 conn(0x7fadbc07a4f0 msgr2=0x7fadbc07a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.685+0000 7fadc1e00700 1 -- 172.21.15.102:0/2166018569 shutdown_connections 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.685+0000 7fadc1e00700 1 -- 172.21.15.102:0/2166018569 wait complete. 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.686+0000 7fadc1e00700 1 Processor -- start 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.686+0000 7fadc1e00700 1 -- start start 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.686+0000 7fadc1e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadbc075650 0x7fadbc1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.686+0000 7fadc1e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadbc075f70 0x7fadbc1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.686+0000 7fadc1e00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fadbc10fe50 0x7fadbc19bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.686+0000 7fadc1e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fadbc115800 con 0x7fadbc075650 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.686+0000 7fadc1e00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fadbc115680 con 0x7fadbc10fe50 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.686+0000 7fadc1e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fadbc115980 con 0x7fadbc075f70 2024-04-26T20:33:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadba000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadbc075f70 0x7fadbc1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:55.865 INFO:teuthology.orchestra.run.smithi102.stdout:219043332150 2024-04-26T20:33:55.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadbaa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadbc075650 0x7fadbc1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:55.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadba000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadbc075f70 0x7fadbc1a1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:59420/0 (socket says 172.21.15.102:59420) 2024-04-26T20:33:55.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadbaa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadbc075650 0x7fadbc1a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41712/0 (socket says 172.21.15.102:41712) 2024-04-26T20:33:55.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadba000700 1 -- 172.21.15.102:0/854722690 learned_addr learned my addr 172.21.15.102:0/854722690 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:55.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadbb400700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fadbc10fe50 0x7fadbc19bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadba000700 1 -- 172.21.15.102:0/854722690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fadbc10fe50 msgr2=0x7fadbc19bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadba000700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fadbc10fe50 0x7fadbc19bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadba000700 1 -- 172.21.15.102:0/854722690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadbc075650 msgr2=0x7fadbc1a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadba000700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadbc075650 0x7fadbc1a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadba000700 1 -- 172.21.15.102:0/854722690 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadbc19c240 con 0x7fadbc075f70 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.687+0000 7fadba000700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadbc075f70 0x7fadbc1a1e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fada800e970 tx=0x7fada800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.688+0000 7fadc0a00700 1 -- 172.21.15.102:0/854722690 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8004220 con 0x7fadbc075f70 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.688+0000 7fadc0a00700 1 -- 172.21.15.102:0/854722690 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fada80043c0 con 0x7fadbc075f70 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.688+0000 7fadc0a00700 1 -- 172.21.15.102:0/854722690 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8003ce0 con 0x7fadbc075f70 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.688+0000 7fadc1e00700 1 -- 172.21.15.102:0/854722690 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadbc19c530 con 0x7fadbc075f70 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.688+0000 7fadc1e00700 1 -- 172.21.15.102:0/854722690 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadbc1a87f0 con 0x7fadbc075f70 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.689+0000 7fadc1e00700 1 -- 172.21.15.102:0/854722690 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadbc06a110 con 0x7fadbc075f70 2024-04-26T20:33:55.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.691+0000 7fadc0a00700 1 -- 172.21.15.102:0/854722690 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fada800c7e0 con 0x7fadbc075f70 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.692+0000 7fadc0a00700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fada00a3d50 0x7fada00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.693+0000 7fadc0a00700 1 -- 172.21.15.102:0/854722690 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fada80d0a80 con 0x7fadbc075f70 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.693+0000 7fadbaa00700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fada00a3d50 0x7fada00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.693+0000 7fadbaa00700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fada00a3d50 0x7fada00a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fadac009c60 tx=0x7fadac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.697+0000 7fadc0a00700 1 -- 172.21.15.102:0/854722690 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fada80d2e20 con 0x7fadbc075f70 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.943+0000 7fadc1e00700 1 -- 172.21.15.102:0/854722690 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fadbc19cfd0 con 0x7fadbc075f70 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.944+0000 7fadc0a00700 1 -- 172.21.15.102:0/854722690 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fada809cf40 con 0x7fadbc075f70 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.946+0000 7fadc1e00700 1 -- 172.21.15.102:0/854722690 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fada00a3d50 msgr2=0x7fada00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.946+0000 7fadc1e00700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fada00a3d50 0x7fada00a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fadac009c60 tx=0x7fadac009260 comp rx=0 tx=0).stop 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.947+0000 7fadc1e00700 1 -- 172.21.15.102:0/854722690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadbc075f70 msgr2=0x7fadbc1a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.947+0000 7fadc1e00700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadbc075f70 0x7fadbc1a1e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fada800e970 tx=0x7fada800ec80 comp rx=0 tx=0).stop 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.947+0000 7fadc1e00700 1 -- 172.21.15.102:0/854722690 shutdown_connections 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.947+0000 7fadc1e00700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fada00a3d50 0x7fada00a6210 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.947+0000 7fadc1e00700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fadbc10fe50 0x7fadbc19bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:55.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.947+0000 7fadc1e00700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadbc075650 0x7fadbc1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:55.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.947+0000 7fadc1e00700 1 --2- 172.21.15.102:0/854722690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadbc075f70 0x7fadbc1a1e60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:55.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.947+0000 7fadc1e00700 1 -- 172.21.15.102:0/854722690 >> 172.21.15.102:0/854722690 conn(0x7fadbc07a4f0 msgr2=0x7fadbc101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:55.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.947+0000 7fadc1e00700 1 -- 172.21.15.102:0/854722690 shutdown_connections 2024-04-26T20:33:55.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:55 smithi102 ceph-mon[28094]: pgmap v364: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:33:55.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:55 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/854722690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T20:33:55.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:54.947+0000 7fadc1e00700 1 -- 172.21.15.102:0/854722690 wait complete. 2024-04-26T20:33:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:55 smithi112 ceph-mon[32769]: pgmap v364: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:33:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:55 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/854722690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T20:33:56.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:55 smithi130 ceph-mon[32704]: pgmap v364: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:33:56.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:55 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/854722690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T20:33:57.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:57 smithi102 ceph-mon[28094]: pgmap v365: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:33:57.882 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:57 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2279378474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T20:33:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:57 smithi112 ceph-mon[32769]: pgmap v365: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:33:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:57 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2279378474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T20:33:58.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:57 smithi130 ceph-mon[32704]: pgmap v365: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:33:58.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:57 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2279378474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T20:33:58.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.401+0000 7fa626e00700 1 -- 172.21.15.102:0/2568502936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa62010d9e0 msgr2=0x7fa62010fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:58.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.401+0000 7fa626e00700 1 --2- 172.21.15.102:0/2568502936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa62010d9e0 0x7fa62010fe50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa618009f90 tx=0x7fa61800b330 comp rx=0 tx=0).stop 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.401+0000 7fa626e00700 1 -- 172.21.15.102:0/2568502936 shutdown_connections 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.401+0000 7fa626e00700 1 --2- 172.21.15.102:0/2568502936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa62010d9e0 0x7fa62010fe50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.401+0000 7fa626e00700 1 --2- 172.21.15.102:0/2568502936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa620104600 0x7fa6201049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.401+0000 7fa626e00700 1 --2- 172.21.15.102:0/2568502936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa620104f20 0x7fa62010d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.401+0000 7fa626e00700 1 -- 172.21.15.102:0/2568502936 >> 172.21.15.102:0/2568502936 conn(0x7fa6200fddb0 msgr2=0x7fa6201001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.401+0000 7fa626e00700 1 -- 172.21.15.102:0/2568502936 shutdown_connections 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.401+0000 7fa626e00700 1 -- 172.21.15.102:0/2568502936 wait complete. 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.402+0000 7fa626e00700 1 Processor -- start 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.402+0000 7fa626e00700 1 -- start start 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa626e00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa620104600 0x7fa62010c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa626e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa620104f20 0x7fa62010ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa626e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa62010d9e0 0x7fa6201a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa626e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa6201135e0 con 0x7fa620104f20 2024-04-26T20:33:58.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa626e00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa620113460 con 0x7fa620104600 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa626e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa620113760 con 0x7fa62010d9e0 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa625000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa620104f20 0x7fa62010ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa625000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa620104f20 0x7fa62010ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53436/0 (socket says 172.21.15.102:53436) 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa625000700 1 -- 172.21.15.102:0/2584518623 learned_addr learned my addr 172.21.15.102:0/2584518623 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa625a00700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa620104600 0x7fa62010c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.403+0000 7fa626400700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa62010d9e0 0x7fa6201a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa625000700 1 -- 172.21.15.102:0/2584518623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa62010d9e0 msgr2=0x7fa6201a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa625000700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa62010d9e0 0x7fa6201a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa625000700 1 -- 172.21.15.102:0/2584518623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa620104600 msgr2=0x7fa62010c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa625000700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa620104600 0x7fa62010c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa625000700 1 -- 172.21.15.102:0/2584518623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6201a50a0 con 0x7fa620104f20 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa625000700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa620104f20 0x7fa62010ccb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00e970 tx=0x7fa60c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa61e000700 1 -- 172.21.15.102:0/2584518623 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c004a30 con 0x7fa620104f20 2024-04-26T20:33:58.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa626e00700 1 -- 172.21.15.102:0/2584518623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6201a5390 con 0x7fa620104f20 2024-04-26T20:33:58.672 INFO:teuthology.orchestra.run.smithi102.stdout:34359738460 2024-04-26T20:33:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa61e000700 1 -- 172.21.15.102:0/2584518623 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa60c00cc90 con 0x7fa620104f20 2024-04-26T20:33:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa61e000700 1 -- 172.21.15.102:0/2584518623 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c005610 con 0x7fa620104f20 2024-04-26T20:33:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.404+0000 7fa626e00700 1 -- 172.21.15.102:0/2584518623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6201acc10 con 0x7fa620104f20 2024-04-26T20:33:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.405+0000 7fa626e00700 1 -- 172.21.15.102:0/2584518623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa62006a110 con 0x7fa620104f20 2024-04-26T20:33:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.408+0000 7fa61e000700 1 -- 172.21.15.102:0/2584518623 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa60c00c7e0 con 0x7fa620104f20 2024-04-26T20:33:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.409+0000 7fa61e000700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa6080a3cd0 0x7fa6080a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:33:58.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.409+0000 7fa61e000700 1 -- 172.21.15.102:0/2584518623 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa60c0d0c20 con 0x7fa620104f20 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.409+0000 7fa625a00700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa6080a3cd0 0x7fa6080a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.410+0000 7fa625a00700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa6080a3cd0 0x7fa6080a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa614009750 tx=0x7fa614009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.412+0000 7fa61e000700 1 -- 172.21.15.102:0/2584518623 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa60c054060 con 0x7fa620104f20 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.643+0000 7fa626e00700 1 -- 172.21.15.102:0/2584518623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa6201acea0 con 0x7fa620104f20 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.643+0000 7fa61e000700 1 -- 172.21.15.102:0/2584518623 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa60c09d0e0 con 0x7fa620104f20 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 -- 172.21.15.102:0/2584518623 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa6080a3cd0 msgr2=0x7fa6080a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa6080a3cd0 0x7fa6080a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa614009750 tx=0x7fa614009310 comp rx=0 tx=0).stop 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 -- 172.21.15.102:0/2584518623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa620104f20 msgr2=0x7fa62010ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa620104f20 0x7fa62010ccb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00e970 tx=0x7fa60c00ec80 comp rx=0 tx=0).stop 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 -- 172.21.15.102:0/2584518623 shutdown_connections 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa6080a3cd0 0x7fa6080a6190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa62010d9e0 0x7fa6201a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa620104600 0x7fa62010c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:58.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 --2- 172.21.15.102:0/2584518623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa620104f20 0x7fa62010ccb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:33:58.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 -- 172.21.15.102:0/2584518623 >> 172.21.15.102:0/2584518623 conn(0x7fa6200fddb0 msgr2=0x7fa62010ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:33:58.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.649+0000 7fa626e00700 1 -- 172.21.15.102:0/2584518623 shutdown_connections 2024-04-26T20:33:58.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:58.650+0000 7fa626e00700 1 -- 172.21.15.102:0/2584518623 wait complete. 2024-04-26T20:33:59.009 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:58 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2584518623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T20:33:59.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:58 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2584518623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T20:33:59.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:58 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2584518623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T20:33:59.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:33:59 smithi112 ceph-mon[32769]: pgmap v366: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:33:59.800 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:33:59 smithi102 ceph-mon[28094]: pgmap v366: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:34:00.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:33:59 smithi130 ceph-mon[32704]: pgmap v366: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:34:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.799+0000 7fa4b8e00700 1 -- 172.21.15.102:0/4018105312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b406b130 msgr2=0x7fa4b406b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.799+0000 7fa4b8e00700 1 --2- 172.21.15.102:0/4018105312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b406b130 0x7fa4b406b510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009990 tx=0x7fa4a4009ca0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.799+0000 7fa4b8e00700 1 -- 172.21.15.102:0/4018105312 shutdown_connections 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.799+0000 7fa4b8e00700 1 --2- 172.21.15.102:0/4018105312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4b410fed0 0x7fa4b41122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.799+0000 7fa4b8e00700 1 --2- 172.21.15.102:0/4018105312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b406b130 0x7fa4b406b510 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.799+0000 7fa4b8e00700 1 --2- 172.21.15.102:0/4018105312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4b406ba50 0x7fa4b410f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.799+0000 7fa4b8e00700 1 -- 172.21.15.102:0/4018105312 >> 172.21.15.102:0/4018105312 conn(0x7fa4b40792d0 msgr2=0x7fa4b40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.799+0000 7fa4b8e00700 1 -- 172.21.15.102:0/4018105312 shutdown_connections 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.799+0000 7fa4b8e00700 1 -- 172.21.15.102:0/4018105312 wait complete. 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.800+0000 7fa4b8e00700 1 Processor -- start 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b8e00700 1 -- start start 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b8e00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4b406b130 0x7fa4b41a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b8e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b406ba50 0x7fa4b41a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b8e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4b410fed0 0x7fa4b419bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:00.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b8e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b41157e0 con 0x7fa4b406ba50 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b8e00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b4115660 con 0x7fa4b406b130 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b8e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b4115960 con 0x7fa4b410fed0 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b1400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b406ba50 0x7fa4b41a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b1e00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4b406b130 0x7fa4b41a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b2800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4b410fed0 0x7fa4b419bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b1e00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4b406b130 0x7fa4b41a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:56722/0 (socket says 172.21.15.102:56722) 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b1e00700 1 -- 172.21.15.102:0/3383738374 learned_addr learned my addr 172.21.15.102:0/3383738374 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.801+0000 7fa4b1400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b406ba50 0x7fa4b41a1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53464/0 (socket says 172.21.15.102:53464) 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4b2800700 1 -- 172.21.15.102:0/3383738374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4b406b130 msgr2=0x7fa4b41a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4b2800700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4b406b130 0x7fa4b41a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4b2800700 1 -- 172.21.15.102:0/3383738374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b406ba50 msgr2=0x7fa4b41a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4b2800700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b406ba50 0x7fa4b41a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4b2800700 1 -- 172.21.15.102:0/3383738374 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b419c2a0 con 0x7fa4b410fed0 2024-04-26T20:34:00.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4b1e00700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4b406b130 0x7fa4b41a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4b2800700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4b410fed0 0x7fa4b419bb40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00ef90 tx=0x7fa4ac00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4aaa00700 1 -- 172.21.15.102:0/3383738374 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0092e0 con 0x7fa4b410fed0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4aaa00700 1 -- 172.21.15.102:0/3383738374 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4ac004bb0 con 0x7fa4b410fed0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4b8e00700 1 -- 172.21.15.102:0/3383738374 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b419c530 con 0x7fa4b410fed0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4aaa00700 1 -- 172.21.15.102:0/3383738374 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac004020 con 0x7fa4b410fed0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.802+0000 7fa4b8e00700 1 -- 172.21.15.102:0/3383738374 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b41a8800 con 0x7fa4b410fed0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.803+0000 7fa4b8e00700 1 -- 172.21.15.102:0/3383738374 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b406a110 con 0x7fa4b410fed0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.805+0000 7fa4aaa00700 1 -- 172.21.15.102:0/3383738374 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa4ac01b030 con 0x7fa4b410fed0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.807+0000 7fa4aaa00700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4980a3cd0 0x7fa4980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.807+0000 7fa4aaa00700 1 -- 172.21.15.102:0/3383738374 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa4ac0d0760 con 0x7fa4b410fed0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.807+0000 7fa4b1e00700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4980a3cd0 0x7fa4980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.808+0000 7fa4b1e00700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4980a3cd0 0x7fa4980a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4006010 tx=0x7fa4a4009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:59.809+0000 7fa4aaa00700 1 -- 172.21.15.102:0/3383738374 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa4ac022020 con 0x7fa4b410fed0 2024-04-26T20:34:00.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.045+0000 7fa4b8e00700 1 -- 172.21.15.102:0/3383738374 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa4b402e490 con 0x7fa4b410fed0 2024-04-26T20:34:00.295 INFO:teuthology.orchestra.run.smithi102.stdout:141733920841 2024-04-26T20:34:00.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.045+0000 7fa4aaa00700 1 -- 172.21.15.102:0/3383738374 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa4ac010040 con 0x7fa4b410fed0 2024-04-26T20:34:00.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 -- 172.21.15.102:0/3383738374 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4980a3cd0 msgr2=0x7fa4980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4980a3cd0 0x7fa4980a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4006010 tx=0x7fa4a4009510 comp rx=0 tx=0).stop 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 -- 172.21.15.102:0/3383738374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4b410fed0 msgr2=0x7fa4b419bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4b410fed0 0x7fa4b419bb40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00ef90 tx=0x7fa4ac00c3a0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 -- 172.21.15.102:0/3383738374 shutdown_connections 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa4980a3cd0 0x7fa4980a6190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4b410fed0 0x7fa4b419bb40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4b406b130 0x7fa4b41a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 --2- 172.21.15.102:0/3383738374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4b406ba50 0x7fa4b41a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 -- 172.21.15.102:0/3383738374 >> 172.21.15.102:0/3383738374 conn(0x7fa4b40792d0 msgr2=0x7fa4b41113c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 -- 172.21.15.102:0/3383738374 shutdown_connections 2024-04-26T20:34:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:00.048+0000 7fa4b8e00700 1 -- 172.21.15.102:0/3383738374 wait complete. 2024-04-26T20:34:00.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:00 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3383738374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T20:34:00.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:00 smithi102 ceph-mon[28094]: pgmap v367: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-26T20:34:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:00 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3383738374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T20:34:01.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:00 smithi112 ceph-mon[32769]: pgmap v367: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-26T20:34:01.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:00 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3383738374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T20:34:01.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:00 smithi130 ceph-mon[32704]: pgmap v367: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-26T20:34:01.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.367+0000 7f6666000700 1 -- 172.21.15.102:0/1739081449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6660075f70 msgr2=0x7f66601140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:01.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.367+0000 7f6666000700 1 --2- 172.21.15.102:0/1739081449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6660075f70 0x7f66601140b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f664c009990 tx=0x7f664c009ca0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.368+0000 7f6666000700 1 -- 172.21.15.102:0/1739081449 shutdown_connections 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.368+0000 7f6666000700 1 --2- 172.21.15.102:0/1739081449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66601145f0 0x7f66601169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.368+0000 7f6666000700 1 --2- 172.21.15.102:0/1739081449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6660075650 0x7f6660075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.368+0000 7f6666000700 1 --2- 172.21.15.102:0/1739081449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6660075f70 0x7f66601140b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.368+0000 7f6666000700 1 -- 172.21.15.102:0/1739081449 >> 172.21.15.102:0/1739081449 conn(0x7f666007a580 msgr2=0x7f666007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.368+0000 7f6666000700 1 -- 172.21.15.102:0/1739081449 shutdown_connections 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.368+0000 7f6666000700 1 -- 172.21.15.102:0/1739081449 wait complete. 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.369+0000 7f6666000700 1 Processor -- start 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.369+0000 7f6666000700 1 -- start start 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.369+0000 7f6666000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660075650 0x7f66601a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.369+0000 7f6666000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6660075f70 0x7f66601a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.369+0000 7f6666000700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66601145f0 0x7f66601a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.369+0000 7f6666000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6660119be0 con 0x7f6660075650 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.369+0000 7f6666000700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6660119a60 con 0x7f66601145f0 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.369+0000 7f6666000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6660119d60 con 0x7f6660075f70 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660075650 0x7f66601a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6660075f70 0x7f66601a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660075650 0x7f66601a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53432/0 (socket says 172.21.15.102:53432) 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6660075f70 0x7f66601a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:59622/0 (socket says 172.21.15.102:59622) 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665f400700 1 -- 172.21.15.102:0/2279378474 learned_addr learned my addr 172.21.15.102:0/2279378474 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665fe00700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66601145f0 0x7f66601a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665ea00700 1 -- 172.21.15.102:0/2279378474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66601145f0 msgr2=0x7f66601a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665ea00700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66601145f0 0x7f66601a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665ea00700 1 -- 172.21.15.102:0/2279378474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660075650 msgr2=0x7f66601a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665ea00700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660075650 0x7f66601a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665ea00700 1 -- 172.21.15.102:0/2279378474 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66601a5220 con 0x7f6660075f70 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.370+0000 7f665ea00700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6660075f70 0x7f66601a0750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f664c006010 tx=0x7f664c0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.371+0000 7f6657e00700 1 -- 172.21.15.102:0/2279378474 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c027070 con 0x7f6660075f70 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.371+0000 7f6666000700 1 -- 172.21.15.102:0/2279378474 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66601a54b0 con 0x7f6660075f70 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.371+0000 7f6657e00700 1 -- 172.21.15.102:0/2279378474 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f664c022470 con 0x7f6660075f70 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.371+0000 7f6657e00700 1 -- 172.21.15.102:0/2279378474 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c017650 con 0x7f6660075f70 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.371+0000 7f6666000700 1 -- 172.21.15.102:0/2279378474 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66601acd90 con 0x7f6660075f70 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.372+0000 7f6666000700 1 -- 172.21.15.102:0/2279378474 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f666006a110 con 0x7f6660075f70 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.374+0000 7f6657e00700 1 -- 172.21.15.102:0/2279378474 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f664c02a070 con 0x7f6660075f70 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.375+0000 7f6657e00700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66440a3d50 0x7f66440a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.376+0000 7f6657e00700 1 -- 172.21.15.102:0/2279378474 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f664c0d13c0 con 0x7f6660075f70 2024-04-26T20:34:01.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.376+0000 7f665f400700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66440a3d50 0x7f66440a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:01.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.376+0000 7f665f400700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66440a3d50 0x7f66440a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6650009d20 tx=0x7f6650009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:01.260 INFO:teuthology.orchestra.run.smithi102.stdout:107374182477 2024-04-26T20:34:01.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.380+0000 7f6657e00700 1 -- 172.21.15.102:0/2279378474 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f664c013070 con 0x7f6660075f70 2024-04-26T20:34:01.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.611+0000 7f6666000700 1 -- 172.21.15.102:0/2279378474 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f66601acf60 con 0x7f6660075f70 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.612+0000 7f6657e00700 1 -- 172.21.15.102:0/2279378474 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f664c09d970 con 0x7f6660075f70 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 -- 172.21.15.102:0/2279378474 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66440a3d50 msgr2=0x7f66440a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66440a3d50 0x7f66440a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6650009d20 tx=0x7f6650009260 comp rx=0 tx=0).stop 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 -- 172.21.15.102:0/2279378474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6660075f70 msgr2=0x7f66601a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6660075f70 0x7f66601a0750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f664c006010 tx=0x7f664c0057c0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 -- 172.21.15.102:0/2279378474 shutdown_connections 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66440a3d50 0x7f66440a6210 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66601145f0 0x7f66601a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6660075650 0x7f66601a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 --2- 172.21.15.102:0/2279378474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6660075f70 0x7f66601a0750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 -- 172.21.15.102:0/2279378474 >> 172.21.15.102:0/2279378474 conn(0x7f666007a580 msgr2=0x7f6660113490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 -- 172.21.15.102:0/2279378474 shutdown_connections 2024-04-26T20:34:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:33:57.615+0000 7f6666000700 1 -- 172.21.15.102:0/2279378474 wait complete. 2024-04-26T20:34:01.327 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332146 got 219043332150 for osd.7 2024-04-26T20:34:01.327 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:02.067 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738454 got 34359738460 for osd.0 2024-04-26T20:34:02.067 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.887+0000 7f39cf800700 1 -- 172.21.15.102:0/3654035788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c810fa50 msgr2=0x7f39c810fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.887+0000 7f39cf800700 1 --2- 172.21.15.102:0/3654035788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c810fa50 0x7f39c810fe30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0098f0 tx=0x7f39bc009c00 comp rx=0 tx=0).stop 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.887+0000 7f39cf800700 1 -- 172.21.15.102:0/3654035788 shutdown_connections 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.887+0000 7f39cf800700 1 --2- 172.21.15.102:0/3654035788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c810a410 0x7f39c8116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.887+0000 7f39cf800700 1 --2- 172.21.15.102:0/3654035788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c810fa50 0x7f39c810fe30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.887+0000 7f39cf800700 1 --2- 172.21.15.102:0/3654035788 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c8109a50 0x7f39c8109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.887+0000 7f39cf800700 1 -- 172.21.15.102:0/3654035788 >> 172.21.15.102:0/3654035788 conn(0x7f39c8076bf0 msgr2=0x7f39c8108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.887+0000 7f39cf800700 1 -- 172.21.15.102:0/3654035788 shutdown_connections 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.887+0000 7f39cf800700 1 -- 172.21.15.102:0/3654035788 wait complete. 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.888+0000 7f39cf800700 1 Processor -- start 2024-04-26T20:34:02.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.889+0000 7f39cf800700 1 -- start start 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.889+0000 7f39cf800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c8109a50 0x7f39c81a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.889+0000 7f39cf800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c810a410 0x7f39c81a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.889+0000 7f39cf800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c810fa50 0x7f39c81a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.889+0000 7f39cf800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8119ca0 con 0x7f39c8109a50 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.889+0000 7f39cf800700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8119b20 con 0x7f39c810a410 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.889+0000 7f39cf800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8119e20 con 0x7f39c810fa50 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39cd800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c810fa50 0x7f39c81a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39cce00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c8109a50 0x7f39c81a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39cd800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c810fa50 0x7f39c81a4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:59678/0 (socket says 172.21.15.102:59678) 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39cd800700 1 -- 172.21.15.102:0/2942675796 learned_addr learned my addr 172.21.15.102:0/2942675796 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39c7e00700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c810a410 0x7f39c81a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39cd800700 1 -- 172.21.15.102:0/2942675796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c810a410 msgr2=0x7f39c81a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39cd800700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c810a410 0x7f39c81a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39cd800700 1 -- 172.21.15.102:0/2942675796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c8109a50 msgr2=0x7f39c81a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39cd800700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c8109a50 0x7f39c81a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39cd800700 1 -- 172.21.15.102:0/2942675796 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c81a5270 con 0x7f39c810fa50 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.890+0000 7f39cd800700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c810fa50 0x7f39c81a4b30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f39c000c370 tx=0x7f39c000c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.891+0000 7f39c5600700 1 -- 172.21.15.102:0/2942675796 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c00092e0 con 0x7f39c810fa50 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.891+0000 7f39c5600700 1 -- 172.21.15.102:0/2942675796 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39c0003a00 con 0x7f39c810fa50 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.891+0000 7f39cf800700 1 -- 172.21.15.102:0/2942675796 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c81a5500 con 0x7f39c810fa50 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.891+0000 7f39c5600700 1 -- 172.21.15.102:0/2942675796 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c00046f0 con 0x7f39c810fa50 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.891+0000 7f39cf800700 1 -- 172.21.15.102:0/2942675796 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c81acde0 con 0x7f39c810fa50 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.892+0000 7f39cf800700 1 -- 172.21.15.102:0/2942675796 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a8005330 con 0x7f39c810fa50 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.894+0000 7f39c5600700 1 -- 172.21.15.102:0/2942675796 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f39c001a020 con 0x7f39c810fa50 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.895+0000 7f39c5600700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f39b00a3cd0 0x7f39b00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.895+0000 7f39c5600700 1 -- 172.21.15.102:0/2942675796 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f39c00cfff0 con 0x7f39c810fa50 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.896+0000 7f39cce00700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f39b00a3cd0 0x7f39b00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.896+0000 7f39cce00700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f39b00a3cd0 0x7f39b00a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0098c0 tx=0x7f39bc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:02.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:02.900+0000 7f39c5600700 1 -- 172.21.15.102:0/2942675796 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f39c0010040 con 0x7f39c810fa50 2024-04-26T20:34:03.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.129+0000 7f39cf800700 1 -- 172.21.15.102:0/2942675796 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f39a80058f0 con 0x7f39c810fa50 2024-04-26T20:34:03.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.129+0000 7f39c5600700 1 -- 172.21.15.102:0/2942675796 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f39c009c4f0 con 0x7f39c810fa50 2024-04-26T20:34:03.131 INFO:teuthology.orchestra.run.smithi102.stdout:163208757315 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.132+0000 7f39cf800700 1 -- 172.21.15.102:0/2942675796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f39b00a3cd0 msgr2=0x7f39b00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.132+0000 7f39cf800700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f39b00a3cd0 0x7f39b00a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0098c0 tx=0x7f39bc019040 comp rx=0 tx=0).stop 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.132+0000 7f39cf800700 1 -- 172.21.15.102:0/2942675796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c810fa50 msgr2=0x7f39c81a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.132+0000 7f39cf800700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c810fa50 0x7f39c81a4b30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f39c000c370 tx=0x7f39c000c680 comp rx=0 tx=0).stop 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.133+0000 7f39cf800700 1 -- 172.21.15.102:0/2942675796 shutdown_connections 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.133+0000 7f39cf800700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f39b00a3cd0 0x7f39b00a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.133+0000 7f39cf800700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39c810fa50 0x7f39c81a4b30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.133+0000 7f39cf800700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c8109a50 0x7f39c81a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.133+0000 7f39cf800700 1 --2- 172.21.15.102:0/2942675796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39c810a410 0x7f39c81a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.133+0000 7f39cf800700 1 -- 172.21.15.102:0/2942675796 >> 172.21.15.102:0/2942675796 conn(0x7f39c8076bf0 msgr2=0x7f39c8077310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.133+0000 7f39cf800700 1 -- 172.21.15.102:0/2942675796 shutdown_connections 2024-04-26T20:34:03.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:03.133+0000 7f39cf800700 1 -- 172.21.15.102:0/2942675796 wait complete. 2024-04-26T20:34:03.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:03 smithi130 ceph-mon[32704]: pgmap v368: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-26T20:34:03.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:03 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2942675796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T20:34:03.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:03 smithi102 ceph-mon[28094]: pgmap v368: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-26T20:34:03.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:03 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2942675796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T20:34:04.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:03 smithi112 ceph-mon[32769]: pgmap v368: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-26T20:34:04.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:03 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2942675796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T20:34:04.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.569+0000 7fbad7200700 1 -- 172.21.15.102:0/3600594545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbad010b620 msgr2=0x7fbad010ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:04.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.569+0000 7fbad7200700 1 --2- 172.21.15.102:0/3600594545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbad010b620 0x7fbad010ba00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbac40098f0 tx=0x7fbac4009c00 comp rx=0 tx=0).stop 2024-04-26T20:34:04.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.569+0000 7fbad7200700 1 -- 172.21.15.102:0/3600594545 shutdown_connections 2024-04-26T20:34:04.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.569+0000 7fbad7200700 1 --2- 172.21.15.102:0/3600594545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbad0105f30 0x7fbad01123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:04.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.569+0000 7fbad7200700 1 --2- 172.21.15.102:0/3600594545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbad010b620 0x7fbad010ba00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:04.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.569+0000 7fbad7200700 1 --2- 172.21.15.102:0/3600594545 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbad0105570 0x7fbad01059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.569+0000 7fbad7200700 1 -- 172.21.15.102:0/3600594545 >> 172.21.15.102:0/3600594545 conn(0x7fbad0101040 msgr2=0x7fbad0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.570+0000 7fbad7200700 1 -- 172.21.15.102:0/3600594545 shutdown_connections 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.570+0000 7fbad7200700 1 -- 172.21.15.102:0/3600594545 wait complete. 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.571+0000 7fbad7200700 1 Processor -- start 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.571+0000 7fbad7200700 1 -- start start 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad7200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbad0105570 0x7fbad01a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad7200700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbad0105f30 0x7fbad01a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad7200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbad010b620 0x7fbad019b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad7200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbad0115770 con 0x7fbad0105570 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad7200700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbad01155f0 con 0x7fbad0105f30 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad7200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbad01158f0 con 0x7fbad010b620 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad5e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbad0105570 0x7fbad01a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:04.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad6800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbad010b620 0x7fbad019b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad5e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbad0105570 0x7fbad01a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53506/0 (socket says 172.21.15.102:53506) 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad6800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbad010b620 0x7fbad019b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:59706/0 (socket says 172.21.15.102:59706) 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad5e00700 1 -- 172.21.15.102:0/4153929665 learned_addr learned my addr 172.21.15.102:0/4153929665 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad5400700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbad0105f30 0x7fbad01a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad5e00700 1 -- 172.21.15.102:0/4153929665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbad010b620 msgr2=0x7fbad019b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad5e00700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbad010b620 0x7fbad019b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad5e00700 1 -- 172.21.15.102:0/4153929665 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbad0105f30 msgr2=0x7fbad01a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad5e00700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbad0105f30 0x7fbad01a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad5e00700 1 -- 172.21.15.102:0/4153929665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad019c130 con 0x7fbad0105570 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.572+0000 7fbad6800700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbad010b620 0x7fbad019b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.573+0000 7fbad5e00700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbad0105570 0x7fbad01a1810 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009230 tx=0x7fbac400fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.573+0000 7fbacaa00700 1 -- 172.21.15.102:0/4153929665 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac40050c0 con 0x7fbad0105570 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.573+0000 7fbad7200700 1 -- 172.21.15.102:0/4153929665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad019c3c0 con 0x7fbad0105570 2024-04-26T20:34:04.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.573+0000 7fbacaa00700 1 -- 172.21.15.102:0/4153929665 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbac400b560 con 0x7fbad0105570 2024-04-26T20:34:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.573+0000 7fbacaa00700 1 -- 172.21.15.102:0/4153929665 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4003a00 con 0x7fbad0105570 2024-04-26T20:34:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.573+0000 7fbad7200700 1 -- 172.21.15.102:0/4153929665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad01a86e0 con 0x7fbad0105570 2024-04-26T20:34:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.574+0000 7fbad7200700 1 -- 172.21.15.102:0/4153929665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad010dab0 con 0x7fbad0105570 2024-04-26T20:34:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.576+0000 7fbacaa00700 1 -- 172.21.15.102:0/4153929665 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fbac400b710 con 0x7fbad0105570 2024-04-26T20:34:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.578+0000 7fbacaa00700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbab80a3cd0 0x7fbab80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.578+0000 7fbacaa00700 1 -- 172.21.15.102:0/4153929665 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbac40d1810 con 0x7fbad0105570 2024-04-26T20:34:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.578+0000 7fbad5400700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbab80a3cd0 0x7fbab80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.579+0000 7fbad5400700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbab80a3cd0 0x7fbab80a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009680 tx=0x7fbac0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:04.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.582+0000 7fbacaa00700 1 -- 172.21.15.102:0/4153929665 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbac4013030 con 0x7fbad0105570 2024-04-26T20:34:04.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.817+0000 7fbad7200700 1 -- 172.21.15.102:0/4153929665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fbad019cf00 con 0x7fbad0105570 2024-04-26T20:34:04.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.818+0000 7fbacaa00700 1 -- 172.21.15.102:0/4153929665 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbac4021380 con 0x7fbad0105570 2024-04-26T20:34:04.819 INFO:teuthology.orchestra.run.smithi102.stdout:253403070515 2024-04-26T20:34:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 -- 172.21.15.102:0/4153929665 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbab80a3cd0 msgr2=0x7fbab80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbab80a3cd0 0x7fbab80a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009680 tx=0x7fbac0009260 comp rx=0 tx=0).stop 2024-04-26T20:34:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 -- 172.21.15.102:0/4153929665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbad0105570 msgr2=0x7fbad01a1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbad0105570 0x7fbad01a1810 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009230 tx=0x7fbac400fa70 comp rx=0 tx=0).stop 2024-04-26T20:34:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 -- 172.21.15.102:0/4153929665 shutdown_connections 2024-04-26T20:34:04.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fbab80a3cd0 0x7fbab80a6190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:04.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbad010b620 0x7fbad019b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:04.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbad0105570 0x7fbad01a1810 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:04.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 --2- 172.21.15.102:0/4153929665 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbad0105f30 0x7fbad01a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:04.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 -- 172.21.15.102:0/4153929665 >> 172.21.15.102:0/4153929665 conn(0x7fbad0101040 msgr2=0x7fbad0102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:04.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 -- 172.21.15.102:0/4153929665 shutdown_connections 2024-04-26T20:34:04.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:04.821+0000 7fbad7200700 1 -- 172.21.15.102:0/4153929665 wait complete. 2024-04-26T20:34:05.231 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182475 got 107374182477 for osd.3 2024-04-26T20:34:05.231 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:05.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.115+0000 7f98ab200700 1 -- 172.21.15.102:0/3805839630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98ac1035a0 msgr2=0x7f98ac103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:05.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.115+0000 7f98ab200700 1 --2- 172.21.15.102:0/3805839630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98ac1035a0 0x7f98ac103980 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f989c0098f0 tx=0x7f989c009c00 comp rx=0 tx=0).stop 2024-04-26T20:34:05.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.115+0000 7f98ab200700 1 -- 172.21.15.102:0/3805839630 shutdown_connections 2024-04-26T20:34:05.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.115+0000 7f98ab200700 1 --2- 172.21.15.102:0/3805839630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98ac108680 0x7f98ac110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:05.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.115+0000 7f98ab200700 1 --2- 172.21.15.102:0/3805839630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98ac1035a0 0x7f98ac103980 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.115+0000 7f98ab200700 1 --2- 172.21.15.102:0/3805839630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98ac103f50 0x7f98ac107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.115+0000 7f98ab200700 1 -- 172.21.15.102:0/3805839630 >> 172.21.15.102:0/3805839630 conn(0x7f98ac0fee30 msgr2=0x7f98ac101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.115+0000 7f98ab200700 1 -- 172.21.15.102:0/3805839630 shutdown_connections 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.115+0000 7f98ab200700 1 -- 172.21.15.102:0/3805839630 wait complete. 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.117+0000 7f98ab200700 1 Processor -- start 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.117+0000 7f98ab200700 1 -- start start 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98ab200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98ac1035a0 0x7f98ac19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98ab200700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98ac103f50 0x7f98ac19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98ab200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98ac108680 0x7f98ac199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98ab200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac113510 con 0x7f98ac1035a0 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98ab200700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac113390 con 0x7f98ac103f50 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98ab200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac113690 con 0x7f98ac108680 2024-04-26T20:34:05.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98a9e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98ac1035a0 0x7f98ac19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98a9e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98ac1035a0 0x7f98ac19f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46830/0 (socket says 172.21.15.102:46830) 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98a9e00700 1 -- 172.21.15.102:0/3252785358 learned_addr learned my addr 172.21.15.102:0/3252785358 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98a9400700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98ac103f50 0x7f98ac19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.118+0000 7f98aa800700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98ac108680 0x7f98ac199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.119+0000 7f98aa800700 1 -- 172.21.15.102:0/3252785358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98ac103f50 msgr2=0x7f98ac19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.119+0000 7f98aa800700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98ac103f50 0x7f98ac19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.119+0000 7f98aa800700 1 -- 172.21.15.102:0/3252785358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98ac1035a0 msgr2=0x7f98ac19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.119+0000 7f98aa800700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98ac1035a0 0x7f98ac19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.119+0000 7f98aa800700 1 -- 172.21.15.102:0/3252785358 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98ac19a070 con 0x7f98ac108680 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.119+0000 7f98a9400700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98ac103f50 0x7f98ac19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.119+0000 7f98a9e00700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98ac1035a0 0x7f98ac19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:05.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.119+0000 7f98aa800700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98ac108680 0x7f98ac199910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f98a400efc0 tx=0x7f98a400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.119+0000 7f98a2a00700 1 -- 172.21.15.102:0/3252785358 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a40092e0 con 0x7f98ac108680 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.120+0000 7f98a2a00700 1 -- 172.21.15.102:0/3252785358 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a4004bb0 con 0x7f98ac108680 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.120+0000 7f98ab200700 1 -- 172.21.15.102:0/3252785358 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98ac19a300 con 0x7f98ac108680 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.120+0000 7f98a2a00700 1 -- 172.21.15.102:0/3252785358 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a4004020 con 0x7f98ac108680 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.120+0000 7f98ab200700 1 -- 172.21.15.102:0/3252785358 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98ac1a65d0 con 0x7f98ac108680 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.121+0000 7f98ab200700 1 -- 172.21.15.102:0/3252785358 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98ac06a110 con 0x7f98ac108680 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.123+0000 7f98a2a00700 1 -- 172.21.15.102:0/3252785358 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f98a401b030 con 0x7f98ac108680 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.124+0000 7f98a2a00700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98900a3cd0 0x7f98900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.124+0000 7f98a2a00700 1 -- 172.21.15.102:0/3252785358 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f98a40d0350 con 0x7f98ac108680 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.124+0000 7f98a9e00700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98900a3cd0 0x7f98900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.125+0000 7f98a9e00700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98900a3cd0 0x7f98900a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f989c009680 tx=0x7f989c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:05.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.128+0000 7f98a2a00700 1 -- 172.21.15.102:0/3252785358 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f98a4022020 con 0x7f98ac108680 2024-04-26T20:34:05.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.360+0000 7f98ab200700 1 -- 172.21.15.102:0/3252785358 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f98ac02e490 con 0x7f98ac108680 2024-04-26T20:34:05.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.360+0000 7f98a2a00700 1 -- 172.21.15.102:0/3252785358 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f98a4010040 con 0x7f98ac108680 2024-04-26T20:34:05.362 INFO:teuthology.orchestra.run.smithi102.stdout:77309411414 2024-04-26T20:34:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 -- 172.21.15.102:0/3252785358 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98900a3cd0 msgr2=0x7f98900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98900a3cd0 0x7f98900a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f989c009680 tx=0x7f989c019040 comp rx=0 tx=0).stop 2024-04-26T20:34:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 -- 172.21.15.102:0/3252785358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98ac108680 msgr2=0x7f98ac199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98ac108680 0x7f98ac199910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f98a400efc0 tx=0x7f98a400c3a0 comp rx=0 tx=0).stop 2024-04-26T20:34:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 -- 172.21.15.102:0/3252785358 shutdown_connections 2024-04-26T20:34:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f98900a3cd0 0x7f98900a6190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98ac108680 0x7f98ac199910 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98ac1035a0 0x7f98ac19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 --2- 172.21.15.102:0/3252785358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98ac103f50 0x7f98ac19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 -- 172.21.15.102:0/3252785358 >> 172.21.15.102:0/3252785358 conn(0x7f98ac0fee30 msgr2=0x7f98ac107810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 -- 172.21.15.102:0/3252785358 shutdown_connections 2024-04-26T20:34:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:05.363+0000 7f98ab200700 1 -- 172.21.15.102:0/3252785358 wait complete. 2024-04-26T20:34:05.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:05 smithi102 ceph-mon[28094]: pgmap v369: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T20:34:05.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:05 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/4153929665' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T20:34:05.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:05 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3252785358' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T20:34:05.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:05 smithi130 ceph-mon[32704]: pgmap v369: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T20:34:05.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:05 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/4153929665' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T20:34:05.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:05 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3252785358' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T20:34:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:05 smithi112 ceph-mon[32769]: pgmap v369: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T20:34:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:05 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/4153929665' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T20:34:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:05 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3252785358' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T20:34:07.691 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:07 smithi102 ceph-mon[28094]: pgmap v370: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-26T20:34:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.689+0000 7ff758a00700 1 -- 172.21.15.102:0/509501534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff754102fe0 msgr2=0x7ff75410f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.689+0000 7ff758a00700 1 --2- 172.21.15.102:0/509501534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff754102fe0 0x7ff75410f170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00b0d0 tx=0x7ff74c00b3e0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.689+0000 7ff758a00700 1 -- 172.21.15.102:0/509501534 shutdown_connections 2024-04-26T20:34:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.689+0000 7ff758a00700 1 --2- 172.21.15.102:0/509501534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff754102fe0 0x7ff75410f170 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.689+0000 7ff758a00700 1 --2- 172.21.15.102:0/509501534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff754108450 0x7ff754108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.689+0000 7ff758a00700 1 --2- 172.21.15.102:0/509501534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff754102640 0x7ff754102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.689+0000 7ff758a00700 1 -- 172.21.15.102:0/509501534 >> 172.21.15.102:0/509501534 conn(0x7ff7540790f0 msgr2=0x7ff754079500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:07.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.690+0000 7ff758a00700 1 -- 172.21.15.102:0/509501534 shutdown_connections 2024-04-26T20:34:07.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.690+0000 7ff758a00700 1 -- 172.21.15.102:0/509501534 wait complete. 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.691+0000 7ff758a00700 1 Processor -- start 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.691+0000 7ff758a00700 1 -- start start 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.691+0000 7ff758a00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff754102640 0x7ff7541a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.691+0000 7ff758a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff754102fe0 0x7ff7541a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.691+0000 7ff758a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff754108450 0x7ff75419bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.691+0000 7ff758a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff754112560 con 0x7ff754102fe0 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.691+0000 7ff758a00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff7541123e0 con 0x7ff754102640 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.691+0000 7ff758a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff7541126e0 con 0x7ff754108450 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.692+0000 7ff751400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff754102fe0 0x7ff7541a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.692+0000 7ff752800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff754108450 0x7ff75419bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.692+0000 7ff751400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff754102fe0 0x7ff7541a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46844/0 (socket says 172.21.15.102:46844) 2024-04-26T20:34:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.692+0000 7ff751400700 1 -- 172.21.15.102:0/4288678921 learned_addr learned my addr 172.21.15.102:0/4288678921 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:07.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.692+0000 7ff751e00700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff754102640 0x7ff7541a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:07.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.692+0000 7ff752800700 1 -- 172.21.15.102:0/4288678921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff754102640 msgr2=0x7ff7541a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.692+0000 7ff752800700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff754102640 0x7ff7541a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.692+0000 7ff752800700 1 -- 172.21.15.102:0/4288678921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff754102fe0 msgr2=0x7ff7541a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.692+0000 7ff752800700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff754102fe0 0x7ff7541a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.692+0000 7ff752800700 1 -- 172.21.15.102:0/4288678921 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75419c210 con 0x7ff754108450 2024-04-26T20:34:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.693+0000 7ff752800700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff754108450 0x7ff75419bab0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00b770 tx=0x7ff74c0079a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.693+0000 7ff74aa00700 1 -- 172.21.15.102:0/4288678921 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c01f070 con 0x7ff754108450 2024-04-26T20:34:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.693+0000 7ff74aa00700 1 -- 172.21.15.102:0/4288678921 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff74c004c50 con 0x7ff754108450 2024-04-26T20:34:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.693+0000 7ff758a00700 1 -- 172.21.15.102:0/4288678921 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75419c4a0 con 0x7ff754108450 2024-04-26T20:34:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.693+0000 7ff74aa00700 1 -- 172.21.15.102:0/4288678921 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c01b690 con 0x7ff754108450 2024-04-26T20:34:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.693+0000 7ff758a00700 1 -- 172.21.15.102:0/4288678921 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7541a87c0 con 0x7ff754108450 2024-04-26T20:34:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.694+0000 7ff758a00700 1 -- 172.21.15.102:0/4288678921 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff75406a110 con 0x7ff754108450 2024-04-26T20:34:07.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.696+0000 7ff74aa00700 1 -- 172.21.15.102:0/4288678921 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff74c022070 con 0x7ff754108450 2024-04-26T20:34:07.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.697+0000 7ff74aa00700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff7380a3d50 0x7ff7380a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:07.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.697+0000 7ff74aa00700 1 -- 172.21.15.102:0/4288678921 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff74c00f020 con 0x7ff754108450 2024-04-26T20:34:07.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.698+0000 7ff751e00700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff7380a3d50 0x7ff7380a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:07.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.698+0000 7ff751e00700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff7380a3d50 0x7ff7380a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff75419cbf0 tx=0x7ff744009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:07.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.702+0000 7ff74aa00700 1 -- 172.21.15.102:0/4288678921 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff74c029020 con 0x7ff754108450 2024-04-26T20:34:07.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:07 smithi130 ceph-mon[32704]: pgmap v370: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-26T20:34:07.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.953+0000 7ff758a00700 1 -- 172.21.15.102:0/4288678921 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ff7541a8990 con 0x7ff754108450 2024-04-26T20:34:07.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.954+0000 7ff74aa00700 1 -- 172.21.15.102:0/4288678921 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff74c0233b0 con 0x7ff754108450 2024-04-26T20:34:07.955 INFO:teuthology.orchestra.run.smithi102.stdout:309237645351 2024-04-26T20:34:07.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 -- 172.21.15.102:0/4288678921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff7380a3d50 msgr2=0x7ff7380a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:07.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff7380a3d50 0x7ff7380a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff75419cbf0 tx=0x7ff744009260 comp rx=0 tx=0).stop 2024-04-26T20:34:07.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 -- 172.21.15.102:0/4288678921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff754108450 msgr2=0x7ff75419bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:07.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff754108450 0x7ff75419bab0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00b770 tx=0x7ff74c0079a0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 -- 172.21.15.102:0/4288678921 shutdown_connections 2024-04-26T20:34:07.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff7380a3d50 0x7ff7380a6210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff754108450 0x7ff75419bab0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff754102640 0x7ff7541a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 --2- 172.21.15.102:0/4288678921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff754102fe0 0x7ff7541a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:07.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 -- 172.21.15.102:0/4288678921 >> 172.21.15.102:0/4288678921 conn(0x7ff7540790f0 msgr2=0x7ff75406bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:07.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.957+0000 7ff758a00700 1 -- 172.21.15.102:0/4288678921 shutdown_connections 2024-04-26T20:34:07.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:07.958+0000 7ff758a00700 1 -- 172.21.15.102:0/4288678921 wait complete. 2024-04-26T20:34:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:07 smithi112 ceph-mon[32769]: pgmap v370: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-26T20:34:08.349 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070509 got 253403070515 for osd.8 2024-04-26T20:34:08.349 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:08.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:08 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/4288678921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T20:34:08.903 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920835 got 141733920841 for osd.4 2024-04-26T20:34:08.903 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:09.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:08 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/4288678921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T20:34:09.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:08 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/4288678921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T20:34:09.895 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:09 smithi102 ceph-mon[28094]: pgmap v371: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T20:34:09.898 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757309 got 163208757315 for osd.5 2024-04-26T20:34:09.898 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:10.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:09 smithi112 ceph-mon[32769]: pgmap v371: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T20:34:10.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:09 smithi130 ceph-mon[32704]: pgmap v371: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-26T20:34:10.461 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411409 got 77309411414 for osd.2 2024-04-26T20:34:10.462 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:10.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.562+0000 7f52b1000700 1 -- 172.21.15.102:0/1085302382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ac10b620 msgr2=0x7f52ac10ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:10.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.562+0000 7f52b1000700 1 --2- 172.21.15.102:0/1085302382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ac10b620 0x7f52ac10ba00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f529c0099c0 tx=0x7f529c009cd0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.562+0000 7f52b1000700 1 -- 172.21.15.102:0/1085302382 shutdown_connections 2024-04-26T20:34:10.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.562+0000 7f52b1000700 1 --2- 172.21.15.102:0/1085302382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f52ac105f30 0x7f52ac1123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.562+0000 7f52b1000700 1 --2- 172.21.15.102:0/1085302382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ac10b620 0x7f52ac10ba00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.562+0000 7f52b1000700 1 --2- 172.21.15.102:0/1085302382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52ac105570 0x7f52ac1059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.562+0000 7f52b1000700 1 -- 172.21.15.102:0/1085302382 >> 172.21.15.102:0/1085302382 conn(0x7f52ac101040 msgr2=0x7f52ac103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.562+0000 7f52b1000700 1 -- 172.21.15.102:0/1085302382 shutdown_connections 2024-04-26T20:34:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.562+0000 7f52b1000700 1 -- 172.21.15.102:0/1085302382 wait complete. 2024-04-26T20:34:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.563+0000 7f52b1000700 1 Processor -- start 2024-04-26T20:34:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.564+0000 7f52b1000700 1 -- start start 2024-04-26T20:34:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.564+0000 7f52b1000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ac105570 0x7f52ac1a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.564+0000 7f52b1000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f52ac105f30 0x7f52ac1a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.564+0000 7f52b1000700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52ac10b620 0x7f52ac19b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.564+0000 7f52b1000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac115770 con 0x7f52ac105570 2024-04-26T20:34:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.564+0000 7f52b1000700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac1155f0 con 0x7f52ac10b620 2024-04-26T20:34:10.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.564+0000 7f52b1000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac1158f0 con 0x7f52ac105f30 2024-04-26T20:34:10.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52ab400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ac105570 0x7f52ac1a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52ab400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ac105570 0x7f52ac1a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46860/0 (socket says 172.21.15.102:46860) 2024-04-26T20:34:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52ab400700 1 -- 172.21.15.102:0/493275131 learned_addr learned my addr 172.21.15.102:0/493275131 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52aaa00700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f52ac105f30 0x7f52ac1a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52abe00700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52ac10b620 0x7f52ac19b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52ab400700 1 -- 172.21.15.102:0/493275131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f52ac105f30 msgr2=0x7f52ac1a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52ab400700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f52ac105f30 0x7f52ac1a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52ab400700 1 -- 172.21.15.102:0/493275131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52ac10b620 msgr2=0x7f52ac19b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52ab400700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52ac10b620 0x7f52ac19b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52ab400700 1 -- 172.21.15.102:0/493275131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ac19c0f0 con 0x7f52ac105570 2024-04-26T20:34:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52abe00700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52ac10b620 0x7f52ac19b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:10.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.565+0000 7f52ab400700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ac105570 0x7f52ac1a1780 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f529c006010 tx=0x7f529c005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:10.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.566+0000 7f52a3e00700 1 -- 172.21.15.102:0/493275131 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c026070 con 0x7f52ac105570 2024-04-26T20:34:10.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.566+0000 7f52a3e00700 1 -- 172.21.15.102:0/493275131 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f529c005680 con 0x7f52ac105570 2024-04-26T20:34:10.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.566+0000 7f52b1000700 1 -- 172.21.15.102:0/493275131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52ac19c380 con 0x7f52ac105570 2024-04-26T20:34:10.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.566+0000 7f52a3e00700 1 -- 172.21.15.102:0/493275131 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c00fb20 con 0x7f52ac105570 2024-04-26T20:34:10.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.566+0000 7f52b1000700 1 -- 172.21.15.102:0/493275131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52ac1a8650 con 0x7f52ac105570 2024-04-26T20:34:10.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.567+0000 7f52b1000700 1 -- 172.21.15.102:0/493275131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52ac10dab0 con 0x7f52ac105570 2024-04-26T20:34:10.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.569+0000 7f52a3e00700 1 -- 172.21.15.102:0/493275131 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f529c029070 con 0x7f52ac105570 2024-04-26T20:34:10.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.570+0000 7f52a3e00700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f52900a3cd0 0x7f52900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:10.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.571+0000 7f52a3e00700 1 -- 172.21.15.102:0/493275131 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f529c0d10d0 con 0x7f52ac105570 2024-04-26T20:34:10.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.571+0000 7f52aaa00700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f52900a3cd0 0x7f52900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:10.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.571+0000 7f52aaa00700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f52900a3cd0 0x7f52900a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5298009680 tx=0x7f5298009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:10.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.575+0000 7f52a3e00700 1 -- 172.21.15.102:0/493275131 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f529c05e060 con 0x7f52ac105570 2024-04-26T20:34:10.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.815+0000 7fa55e200700 1 -- 172.21.15.102:0/1638412615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa55810fcd0 msgr2=0x7fa5581120c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:10.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.815+0000 7fa55e200700 1 --2- 172.21.15.102:0/1638412615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa55810fcd0 0x7fa5581120c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa550009f90 tx=0x7fa55000b330 comp rx=0 tx=0).stop 2024-04-26T20:34:10.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.815+0000 7fa55e200700 1 -- 172.21.15.102:0/1638412615 shutdown_connections 2024-04-26T20:34:10.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.815+0000 7fa55e200700 1 --2- 172.21.15.102:0/1638412615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa55810fcd0 0x7fa5581120c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.815+0000 7fa55e200700 1 --2- 172.21.15.102:0/1638412615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558102120 0x7fa558102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.815+0000 7fa55e200700 1 --2- 172.21.15.102:0/1638412615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa558102ad0 0x7fa55810f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.815+0000 7fa55e200700 1 -- 172.21.15.102:0/1638412615 >> 172.21.15.102:0/1638412615 conn(0x7fa5580fdd40 msgr2=0x7fa558100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:10.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.815+0000 7fa55e200700 1 -- 172.21.15.102:0/1638412615 shutdown_connections 2024-04-26T20:34:10.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.815+0000 7fa55e200700 1 -- 172.21.15.102:0/1638412615 wait complete. 2024-04-26T20:34:10.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.816+0000 7fa55e200700 1 Processor -- start 2024-04-26T20:34:10.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55e200700 1 -- start start 2024-04-26T20:34:10.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55e200700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa558102120 0x7fa5581a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:10.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55e200700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558102ad0 0x7fa5581a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:10.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55e200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa55810fcd0 0x7fa55819ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:10.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55e200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581157a0 con 0x7fa55810fcd0 2024-04-26T20:34:10.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55e200700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa558115620 con 0x7fa558102ad0 2024-04-26T20:34:10.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55e200700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa558115920 con 0x7fa558102120 2024-04-26T20:34:10.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa55810fcd0 0x7fa55819ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:10.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55ce00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa558102120 0x7fa5581a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:10.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55d800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa55810fcd0 0x7fa55819ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46874/0 (socket says 172.21.15.102:46874) 2024-04-26T20:34:10.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa557e00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558102ad0 0x7fa5581a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:10.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.817+0000 7fa55d800700 1 -- 172.21.15.102:0/1098227650 learned_addr learned my addr 172.21.15.102:0/1098227650 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:10.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.818+0000 7fa55d800700 1 -- 172.21.15.102:0/1098227650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa558102120 msgr2=0x7fa5581a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:10.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.818+0000 7fa55d800700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa558102120 0x7fa5581a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.818+0000 7fa55d800700 1 -- 172.21.15.102:0/1098227650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558102ad0 msgr2=0x7fa5581a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:10.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.818+0000 7fa55d800700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558102ad0 0x7fa5581a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.818+0000 7fa55d800700 1 -- 172.21.15.102:0/1098227650 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55819c1a0 con 0x7fa55810fcd0 2024-04-26T20:34:10.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.818+0000 7fa55d800700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa55810fcd0 0x7fa55819ba40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa55000b6c0 tx=0x7fa55000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:10.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.818+0000 7fa555600700 1 -- 172.21.15.102:0/1098227650 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550021070 con 0x7fa55810fcd0 2024-04-26T20:34:10.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.818+0000 7fa555600700 1 -- 172.21.15.102:0/1098227650 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa55000ce50 con 0x7fa55810fcd0 2024-04-26T20:34:10.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.818+0000 7fa55e200700 1 -- 172.21.15.102:0/1098227650 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa55819c430 con 0x7fa55810fcd0 2024-04-26T20:34:10.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.819+0000 7fa55e200700 1 -- 172.21.15.102:0/1098227650 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5581a8700 con 0x7fa55810fcd0 2024-04-26T20:34:10.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.819+0000 7fa555600700 1 -- 172.21.15.102:0/1098227650 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550005040 con 0x7fa55810fcd0 2024-04-26T20:34:10.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.820+0000 7fa55e200700 1 -- 172.21.15.102:0/1098227650 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa55806a110 con 0x7fa55810fcd0 2024-04-26T20:34:10.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.822+0000 7fa555600700 1 -- 172.21.15.102:0/1098227650 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa550024070 con 0x7fa55810fcd0 2024-04-26T20:34:10.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.823+0000 7fa555600700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5400a3cd0 0x7fa5400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:10.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.824+0000 7fa555600700 1 -- 172.21.15.102:0/1098227650 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa55000f020 con 0x7fa55810fcd0 2024-04-26T20:34:10.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.824+0000 7fa55ce00700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5400a3cd0 0x7fa5400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:10.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.824+0000 7fa55ce00700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5400a3cd0 0x7fa5400a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa558110490 tx=0x7fa54c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:10.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.826+0000 7fa555600700 1 -- 172.21.15.102:0/1098227650 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa55002b020 con 0x7fa55810fcd0 2024-04-26T20:34:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.833+0000 7f52b1000700 1 -- 172.21.15.102:0/493275131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f52ac19ce70 con 0x7f52ac105570 2024-04-26T20:34:10.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.833+0000 7f52a3e00700 1 -- 172.21.15.102:0/493275131 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f529c013070 con 0x7f52ac105570 2024-04-26T20:34:10.834 INFO:teuthology.orchestra.run.smithi102.stdout:343597383716 2024-04-26T20:34:10.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 -- 172.21.15.102:0/493275131 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f52900a3cd0 msgr2=0x7f52900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:10.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f52900a3cd0 0x7f52900a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5298009680 tx=0x7f5298009260 comp rx=0 tx=0).stop 2024-04-26T20:34:10.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 -- 172.21.15.102:0/493275131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ac105570 msgr2=0x7f52ac1a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:10.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ac105570 0x7f52ac1a1780 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f529c006010 tx=0x7f529c005280 comp rx=0 tx=0).stop 2024-04-26T20:34:10.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 -- 172.21.15.102:0/493275131 shutdown_connections 2024-04-26T20:34:10.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f52900a3cd0 0x7f52900a6190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52ac10b620 0x7f52ac19b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ac105570 0x7f52ac1a1780 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 --2- 172.21.15.102:0/493275131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f52ac105f30 0x7f52ac1a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:10.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 -- 172.21.15.102:0/493275131 >> 172.21.15.102:0/493275131 conn(0x7f52ac101040 msgr2=0x7f52ac102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:10.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 -- 172.21.15.102:0/493275131 shutdown_connections 2024-04-26T20:34:10.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:10.836+0000 7f52b1000700 1 -- 172.21.15.102:0/493275131 wait complete. 2024-04-26T20:34:11.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.073+0000 7fa55e200700 1 -- 172.21.15.102:0/1098227650 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa55802e490 con 0x7fa55810fcd0 2024-04-26T20:34:11.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.074+0000 7fa555600700 1 -- 172.21.15.102:0/1098227650 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa55009e720 con 0x7fa55810fcd0 2024-04-26T20:34:11.075 INFO:teuthology.orchestra.run.smithi102.stdout:197568495679 2024-04-26T20:34:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 -- 172.21.15.102:0/1098227650 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5400a3cd0 msgr2=0x7fa5400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5400a3cd0 0x7fa5400a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa558110490 tx=0x7fa54c009260 comp rx=0 tx=0).stop 2024-04-26T20:34:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 -- 172.21.15.102:0/1098227650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa55810fcd0 msgr2=0x7fa55819ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa55810fcd0 0x7fa55819ba40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa55000b6c0 tx=0x7fa55000cc30 comp rx=0 tx=0).stop 2024-04-26T20:34:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 -- 172.21.15.102:0/1098227650 shutdown_connections 2024-04-26T20:34:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5400a3cd0 0x7fa5400a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa55810fcd0 0x7fa55819ba40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa558102120 0x7fa5581a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 --2- 172.21.15.102:0/1098227650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558102ad0 0x7fa5581a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 -- 172.21.15.102:0/1098227650 >> 172.21.15.102:0/1098227650 conn(0x7fa5580fdd40 msgr2=0x7fa558110e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:11.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.077+0000 7fa55e200700 1 -- 172.21.15.102:0/1098227650 shutdown_connections 2024-04-26T20:34:11.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.078+0000 7fa55e200700 1 -- 172.21.15.102:0/1098227650 wait complete. 2024-04-26T20:34:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.508+0000 7fc39b600700 1 -- 172.21.15.102:0/3920253967 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc39410b690 msgr2=0x7fc39410ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.508+0000 7fc39b600700 1 --2- 172.21.15.102:0/3920253967 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc39410b690 0x7fc39410ba70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc388009990 tx=0x7fc388009ca0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.508+0000 7fc39b600700 1 -- 172.21.15.102:0/3920253967 shutdown_connections 2024-04-26T20:34:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.508+0000 7fc39b600700 1 --2- 172.21.15.102:0/3920253967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc394105fc0 0x7fc3941124d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.508+0000 7fc39b600700 1 --2- 172.21.15.102:0/3920253967 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc39410b690 0x7fc39410ba70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.508+0000 7fc39b600700 1 --2- 172.21.15.102:0/3920253967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc394105600 0x7fc394105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.508+0000 7fc39b600700 1 -- 172.21.15.102:0/3920253967 >> 172.21.15.102:0/3920253967 conn(0x7fc3941010d0 msgr2=0x7fc3941034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.508+0000 7fc39b600700 1 -- 172.21.15.102:0/3920253967 shutdown_connections 2024-04-26T20:34:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.508+0000 7fc39b600700 1 -- 172.21.15.102:0/3920253967 wait complete. 2024-04-26T20:34:11.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.509+0000 7fc39b600700 1 Processor -- start 2024-04-26T20:34:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.509+0000 7fc39b600700 1 -- start start 2024-04-26T20:34:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc39b600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc394105600 0x7fc3941a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc39b600700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc394105fc0 0x7fc3941a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc39b600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc39410b690 0x7fc39419bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc39b600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc394115830 con 0x7fc394105600 2024-04-26T20:34:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc39b600700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc3941156b0 con 0x7fc394105fc0 2024-04-26T20:34:11.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc39b600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc3941159b0 con 0x7fc39410b690 2024-04-26T20:34:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc398c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc394105600 0x7fc3941a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc398c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc394105600 0x7fc3941a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46892/0 (socket says 172.21.15.102:46892) 2024-04-26T20:34:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc398c00700 1 -- 172.21.15.102:0/1046495249 learned_addr learned my addr 172.21.15.102:0/1046495249 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc399600700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc39410b690 0x7fc39419bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc393e00700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc394105fc0 0x7fc3941a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc398c00700 1 -- 172.21.15.102:0/1046495249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc39410b690 msgr2=0x7fc39419bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:11.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc398c00700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc39410b690 0x7fc39419bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc398c00700 1 -- 172.21.15.102:0/1046495249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc394105fc0 msgr2=0x7fc3941a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.510+0000 7fc398c00700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc394105fc0 0x7fc3941a1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.511+0000 7fc398c00700 1 -- 172.21.15.102:0/1046495249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39419c450 con 0x7fc394105600 2024-04-26T20:34:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.511+0000 7fc398c00700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc394105600 0x7fc3941a1a50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc388000c00 tx=0x7fc3880047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.511+0000 7fc391600700 1 -- 172.21.15.102:0/1046495249 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc388026070 con 0x7fc394105600 2024-04-26T20:34:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.511+0000 7fc391600700 1 -- 172.21.15.102:0/1046495249 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc388021410 con 0x7fc394105600 2024-04-26T20:34:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.511+0000 7fc39b600700 1 -- 172.21.15.102:0/1046495249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39419c6e0 con 0x7fc394105600 2024-04-26T20:34:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.511+0000 7fc391600700 1 -- 172.21.15.102:0/1046495249 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc388017650 con 0x7fc394105600 2024-04-26T20:34:11.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.512+0000 7fc39b600700 1 -- 172.21.15.102:0/1046495249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3941a8800 con 0x7fc394105600 2024-04-26T20:34:11.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.513+0000 7fc39b600700 1 -- 172.21.15.102:0/1046495249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc394106d40 con 0x7fc394105600 2024-04-26T20:34:11.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.515+0000 7fc391600700 1 -- 172.21.15.102:0/1046495249 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc388029070 con 0x7fc394105600 2024-04-26T20:34:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.516+0000 7fc391600700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc37c0a3cd0 0x7fc37c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.516+0000 7fc391600700 1 -- 172.21.15.102:0/1046495249 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc3880d15b0 con 0x7fc394105600 2024-04-26T20:34:11.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.517+0000 7fc393e00700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc37c0a3cd0 0x7fc37c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:11.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.517+0000 7fc393e00700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc37c0a3cd0 0x7fc37c0a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc39419d270 tx=0x7fc380009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:11.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.521+0000 7fc391600700 1 -- 172.21.15.102:0/1046495249 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc388013070 con 0x7fc394105600 2024-04-26T20:34:11.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.758+0000 7fc39b600700 1 -- 172.21.15.102:0/1046495249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fc39402e490 con 0x7fc394105600 2024-04-26T20:34:11.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.758+0000 7fc391600700 1 -- 172.21.15.102:0/1046495249 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc38809db00 con 0x7fc394105600 2024-04-26T20:34:11.760 INFO:teuthology.orchestra.run.smithi102.stdout:287762808878 2024-04-26T20:34:11.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.761+0000 7fc39b600700 1 -- 172.21.15.102:0/1046495249 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc37c0a3cd0 msgr2=0x7fc37c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:11.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.761+0000 7fc39b600700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc37c0a3cd0 0x7fc37c0a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc39419d270 tx=0x7fc380009260 comp rx=0 tx=0).stop 2024-04-26T20:34:11.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.761+0000 7fc39b600700 1 -- 172.21.15.102:0/1046495249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc394105600 msgr2=0x7fc3941a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:11.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.761+0000 7fc39b600700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc394105600 0x7fc3941a1a50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc388000c00 tx=0x7fc3880047b0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.763+0000 7fc39b600700 1 -- 172.21.15.102:0/1046495249 shutdown_connections 2024-04-26T20:34:11.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.763+0000 7fc39b600700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc37c0a3cd0 0x7fc37c0a6190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.763+0000 7fc39b600700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc39410b690 0x7fc39419bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.763+0000 7fc39b600700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc394105600 0x7fc3941a1a50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.763+0000 7fc39b600700 1 --2- 172.21.15.102:0/1046495249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc394105fc0 0x7fc3941a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:11.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.763+0000 7fc39b600700 1 -- 172.21.15.102:0/1046495249 >> 172.21.15.102:0/1046495249 conn(0x7fc3941010d0 msgr2=0x7fc3941028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:11.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.763+0000 7fc39b600700 1 -- 172.21.15.102:0/1046495249 shutdown_connections 2024-04-26T20:34:11.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:11.767+0000 7fc39b600700 1 -- 172.21.15.102:0/1046495249 wait complete. 2024-04-26T20:34:11.778 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:11.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:11 smithi102 ceph-mon[28094]: pgmap v372: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-26T20:34:11.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:11 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/493275131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T20:34:11.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:11 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1098227650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T20:34:12.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:11 smithi112 ceph-mon[32769]: pgmap v372: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-26T20:34:12.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:11 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/493275131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T20:34:12.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:11 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1098227650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T20:34:12.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:11 smithi130 ceph-mon[32704]: pgmap v372: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-26T20:34:12.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:11 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/493275131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T20:34:12.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:11 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1098227650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T20:34:12.499 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383709 got 343597383716 for osd.11 2024-04-26T20:34:12.499 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:12.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:12 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1046495249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T20:34:13.023 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:12 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1046495249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T20:34:13.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:12 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1046495249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T20:34:13.298 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808871 got 287762808878 for osd.9 2024-04-26T20:34:13.299 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:13.754 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:13 smithi102 ceph-mon[28094]: pgmap v373: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:34:13.756 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645345 got 309237645351 for osd.10 2024-04-26T20:34:13.756 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:14.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:13 smithi112 ceph-mon[32769]: pgmap v373: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:34:14.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:13 smithi130 ceph-mon[32704]: pgmap v373: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:34:14.574 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495669 got 197568495679 for osd.6 2024-04-26T20:34:14.575 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:15.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.895+0000 7f948ac00700 1 -- 172.21.15.102:0/1501768095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94841094f0 msgr2=0x7f94841098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:15.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.895+0000 7f948ac00700 1 --2- 172.21.15.102:0/1501768095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94841094f0 0x7f94841098d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f94740098f0 tx=0x7f9474009c00 comp rx=0 tx=0).stop 2024-04-26T20:34:15.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.895+0000 7f948ac00700 1 -- 172.21.15.102:0/1501768095 shutdown_connections 2024-04-26T20:34:15.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.895+0000 7f948ac00700 1 --2- 172.21.15.102:0/1501768095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9484103e30 0x7f94841102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:15.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.895+0000 7f948ac00700 1 --2- 172.21.15.102:0/1501768095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94841094f0 0x7f94841098d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:15.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.895+0000 7f948ac00700 1 --2- 172.21.15.102:0/1501768095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9484103470 0x7f94841038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:15.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.895+0000 7f948ac00700 1 -- 172.21.15.102:0/1501768095 >> 172.21.15.102:0/1501768095 conn(0x7f948407a5d0 msgr2=0x7f948407a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:15.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.895+0000 7f948ac00700 1 -- 172.21.15.102:0/1501768095 shutdown_connections 2024-04-26T20:34:15.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.895+0000 7f948ac00700 1 -- 172.21.15.102:0/1501768095 wait complete. 2024-04-26T20:34:15.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.896+0000 7f948ac00700 1 Processor -- start 2024-04-26T20:34:15.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.896+0000 7f948ac00700 1 -- start start 2024-04-26T20:34:15.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.896+0000 7f948ac00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9484103470 0x7f948419be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:15.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.896+0000 7f948ac00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9484103e30 0x7f948419c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:15.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.896+0000 7f948ac00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94841094f0 0x7f94841a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:15.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.896+0000 7f948ac00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9484113640 con 0x7f9484103470 2024-04-26T20:34:15.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.896+0000 7f948ac00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f94841134c0 con 0x7f9484103e30 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.896+0000 7f948ac00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f94841137c0 con 0x7f94841094f0 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.897+0000 7f9483e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9484103470 0x7f948419be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.897+0000 7f9483400700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9484103e30 0x7f948419c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.897+0000 7f9483400700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9484103e30 0x7f948419c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:42164/0 (socket says 172.21.15.102:42164) 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.897+0000 7f9488c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94841094f0 0x7f94841a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.897+0000 7f9483400700 1 -- 172.21.15.102:0/3760382039 learned_addr learned my addr 172.21.15.102:0/3760382039 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.897+0000 7f9483400700 1 -- 172.21.15.102:0/3760382039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94841094f0 msgr2=0x7f94841a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.897+0000 7f9483400700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94841094f0 0x7f94841a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.897+0000 7f9483400700 1 -- 172.21.15.102:0/3760382039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9484103470 msgr2=0x7f948419be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.897+0000 7f9483400700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9484103470 0x7f948419be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:15.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.897+0000 7f9483400700 1 -- 172.21.15.102:0/3760382039 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94841a0e20 con 0x7f9484103e30 2024-04-26T20:34:15.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.898+0000 7f9483400700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9484103e30 0x7f948419c350 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f947000e9a0 tx=0x7f947000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:15.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.898+0000 7f9480c00700 1 -- 172.21.15.102:0/3760382039 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947000ca10 con 0x7f9484103e30 2024-04-26T20:34:15.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.898+0000 7f9480c00700 1 -- 172.21.15.102:0/3760382039 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f947000cbb0 con 0x7f9484103e30 2024-04-26T20:34:15.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.898+0000 7f948ac00700 1 -- 172.21.15.102:0/3760382039 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94841a1110 con 0x7f9484103e30 2024-04-26T20:34:15.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.898+0000 7f9480c00700 1 -- 172.21.15.102:0/3760382039 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470018680 con 0x7f9484103e30 2024-04-26T20:34:15.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.898+0000 7f948ac00700 1 -- 172.21.15.102:0/3760382039 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94841a8990 con 0x7f9484103e30 2024-04-26T20:34:15.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.899+0000 7f948ac00700 1 -- 172.21.15.102:0/3760382039 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f948406a110 con 0x7f9484103e30 2024-04-26T20:34:15.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.904+0000 7f9480c00700 1 -- 172.21.15.102:0/3760382039 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f9470010430 con 0x7f9484103e30 2024-04-26T20:34:15.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.905+0000 7f9480c00700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94680a3cd0 0x7f94680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:15.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.906+0000 7f9480c00700 1 -- 172.21.15.102:0/3760382039 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f94700d0b80 con 0x7f9484103e30 2024-04-26T20:34:15.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.906+0000 7f9483e00700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94680a3cd0 0x7f94680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:15.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.906+0000 7f9480c00700 1 -- 172.21.15.102:0/3760382039 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f947009d040 con 0x7f9484103e30 2024-04-26T20:34:15.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:15.906+0000 7f9483e00700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94680a3cd0 0x7f94680a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f94740098c0 tx=0x7f9474019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:16.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:15 smithi112 ceph-mon[32769]: pgmap v374: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:34:16.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:15 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:34:16.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:15 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:34:16.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:15 smithi130 ceph-mon[32704]: pgmap v374: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:34:16.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:15 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:34:16.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:15 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:34:16.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:15 smithi102 ceph-mon[28094]: pgmap v374: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:34:16.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:34:16.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:15 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:34:16.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.150+0000 7f948ac00700 1 -- 172.21.15.102:0/3760382039 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f94841a8b60 con 0x7f9484103e30 2024-04-26T20:34:16.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.150+0000 7f9480c00700 1 -- 172.21.15.102:0/3760382039 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f947009c940 con 0x7f9484103e30 2024-04-26T20:34:16.151 INFO:teuthology.orchestra.run.smithi102.stdout:55834574940 2024-04-26T20:34:16.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.153+0000 7f948ac00700 1 -- 172.21.15.102:0/3760382039 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94680a3cd0 msgr2=0x7f94680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:16.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.153+0000 7f948ac00700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94680a3cd0 0x7f94680a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f94740098c0 tx=0x7f9474019040 comp rx=0 tx=0).stop 2024-04-26T20:34:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.153+0000 7f948ac00700 1 -- 172.21.15.102:0/3760382039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9484103e30 msgr2=0x7f948419c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.153+0000 7f948ac00700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9484103e30 0x7f948419c350 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f947000e9a0 tx=0x7f947000ecb0 comp rx=0 tx=0).stop 2024-04-26T20:34:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.153+0000 7f948ac00700 1 -- 172.21.15.102:0/3760382039 shutdown_connections 2024-04-26T20:34:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.153+0000 7f948ac00700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f94680a3cd0 0x7f94680a6190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.153+0000 7f948ac00700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94841094f0 0x7f94841a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.153+0000 7f948ac00700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9484103470 0x7f948419be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.153+0000 7f948ac00700 1 --2- 172.21.15.102:0/3760382039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9484103e30 0x7f948419c350 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.153+0000 7f948ac00700 1 -- 172.21.15.102:0/3760382039 >> 172.21.15.102:0/3760382039 conn(0x7f948407a5d0 msgr2=0x7f948410e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.154+0000 7f948ac00700 1 -- 172.21.15.102:0/3760382039 shutdown_connections 2024-04-26T20:34:16.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:16.154+0000 7f948ac00700 1 -- 172.21.15.102:0/3760382039 wait complete. 2024-04-26T20:34:16.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:16 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:34:16.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:16 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:34:16.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:16 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:34:16.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:16 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3760382039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T20:34:16.996 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574934 got 55834574940 for osd.1 2024-04-26T20:34:16.996 DEBUG:teuthology.parallel:result is None 2024-04-26T20:34:16.996 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T20:34:16.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T20:34:17.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:16 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:34:17.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:16 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:34:17.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:16 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:34:17.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:16 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3760382039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T20:34:17.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:16 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:34:17.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:16 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:34:17.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:16 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:34:17.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:16 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3760382039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T20:34:17.434 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:17.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:17 smithi102 ceph-mon[28094]: pgmap v375: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:34:17.928 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:17 smithi112 ceph-mon[32769]: pgmap v375: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:34:18.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:17 smithi130 ceph-mon[32704]: pgmap v375: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:34:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.717+0000 7f6982a00700 1 -- 172.21.15.102:0/810088268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f697c1145f0 msgr2=0x7f697c1169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.717+0000 7f6982a00700 1 --2- 172.21.15.102:0/810088268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f697c1145f0 0x7f697c1169e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6974009990 tx=0x7f6974009ca0 comp rx=0 tx=0).stop 2024-04-26T20:34:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.717+0000 7f6982a00700 1 -- 172.21.15.102:0/810088268 shutdown_connections 2024-04-26T20:34:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.717+0000 7f6982a00700 1 --2- 172.21.15.102:0/810088268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f697c1145f0 0x7f697c1169e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.717+0000 7f6982a00700 1 --2- 172.21.15.102:0/810088268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f697c075650 0x7f697c075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.718+0000 7f6982a00700 1 --2- 172.21.15.102:0/810088268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f697c075f70 0x7f697c1140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.718+0000 7f6982a00700 1 -- 172.21.15.102:0/810088268 >> 172.21.15.102:0/810088268 conn(0x7f697c07a580 msgr2=0x7f697c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.718+0000 7f6982a00700 1 -- 172.21.15.102:0/810088268 shutdown_connections 2024-04-26T20:34:19.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.718+0000 7f6982a00700 1 -- 172.21.15.102:0/810088268 wait complete. 2024-04-26T20:34:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f6982a00700 1 Processor -- start 2024-04-26T20:34:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f6982a00700 1 -- start start 2024-04-26T20:34:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f6982a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f697c075650 0x7f697c1a0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f6982a00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f697c075f70 0x7f697c1a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f6982a00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f697c1145f0 0x7f697c1a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f6982a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f697c119bd0 con 0x7f697c075650 2024-04-26T20:34:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f6982a00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f697c119a50 con 0x7f697c1145f0 2024-04-26T20:34:19.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f6982a00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f697c119d50 con 0x7f697c075f70 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f697be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f697c075650 0x7f697c1a0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f6980a00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f697c1145f0 0x7f697c1a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.719+0000 7f6980a00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f697c1145f0 0x7f697c1a4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:42184/0 (socket says 172.21.15.102:42184) 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.720+0000 7f6980a00700 1 -- 172.21.15.102:0/2560881130 learned_addr learned my addr 172.21.15.102:0/2560881130 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.720+0000 7f6973400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f697c075f70 0x7f697c1a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.720+0000 7f6980a00700 1 -- 172.21.15.102:0/2560881130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f697c075f70 msgr2=0x7f697c1a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.720+0000 7f6980a00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f697c075f70 0x7f697c1a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.720+0000 7f6980a00700 1 -- 172.21.15.102:0/2560881130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f697c075650 msgr2=0x7f697c1a0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.720+0000 7f6980a00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f697c075650 0x7f697c1a0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.720+0000 7f6980a00700 1 -- 172.21.15.102:0/2560881130 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c1a5050 con 0x7f697c1145f0 2024-04-26T20:34:19.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.720+0000 7f6980a00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f697c1145f0 0x7f697c1a4910 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f69740047b0 tx=0x7f697400b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.721+0000 7f6979600700 1 -- 172.21.15.102:0/2560881130 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974027070 con 0x7f697c1145f0 2024-04-26T20:34:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.721+0000 7f6982a00700 1 -- 172.21.15.102:0/2560881130 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697c1a52e0 con 0x7f697c1145f0 2024-04-26T20:34:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.721+0000 7f6979600700 1 -- 172.21.15.102:0/2560881130 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6974022470 con 0x7f697c1145f0 2024-04-26T20:34:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.721+0000 7f6979600700 1 -- 172.21.15.102:0/2560881130 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974017650 con 0x7f697c1145f0 2024-04-26T20:34:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.721+0000 7f6982a00700 1 -- 172.21.15.102:0/2560881130 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f697c1acbc0 con 0x7f697c1145f0 2024-04-26T20:34:19.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.722+0000 7f6982a00700 1 -- 172.21.15.102:0/2560881130 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f697c076d40 con 0x7f697c1145f0 2024-04-26T20:34:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.724+0000 7f6979600700 1 -- 172.21.15.102:0/2560881130 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f697402a070 con 0x7f697c1145f0 2024-04-26T20:34:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.725+0000 7f6979600700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f69600a3cd0 0x7f69600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.725+0000 7f6979600700 1 -- 172.21.15.102:0/2560881130 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f69740d1770 con 0x7f697c1145f0 2024-04-26T20:34:19.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.725+0000 7f697be00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f69600a3cd0 0x7f69600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.726+0000 7f697be00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f69600a3cd0 0x7f69600a6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f696c00f540 tx=0x7f696c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:19.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.729+0000 7f6979600700 1 -- 172.21.15.102:0/2560881130 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f697405f060 con 0x7f697c1145f0 2024-04-26T20:34:19.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:19 smithi102 ceph-mon[28094]: pgmap v376: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:34:19.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.954+0000 7f6982a00700 1 -- 172.21.15.102:0/2560881130 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f697c1107a0 con 0x7f69600a3cd0 2024-04-26T20:34:19.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.986+0000 7f6979600700 1 -- 172.21.15.102:0/2560881130 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340388 (secure 0 0 0) 0x7f697c1107a0 con 0x7f69600a3cd0 2024-04-26T20:34:19.995 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:19.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 -- 172.21.15.102:0/2560881130 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f69600a3cd0 msgr2=0x7f69600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:19.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f69600a3cd0 0x7f69600a6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f696c00f540 tx=0x7f696c00b040 comp rx=0 tx=0).stop 2024-04-26T20:34:19.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 -- 172.21.15.102:0/2560881130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f697c1145f0 msgr2=0x7f697c1a4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:19.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f697c1145f0 0x7f697c1a4910 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f69740047b0 tx=0x7f697400b610 comp rx=0 tx=0).stop 2024-04-26T20:34:19.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 -- 172.21.15.102:0/2560881130 shutdown_connections 2024-04-26T20:34:19.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f69600a3cd0 0x7f69600a6190 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:19.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f697c1145f0 0x7f697c1a4910 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:19.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f697c075650 0x7f697c1a0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:19.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 --2- 172.21.15.102:0/2560881130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f697c075f70 0x7f697c1a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:19.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 -- 172.21.15.102:0/2560881130 >> 172.21.15.102:0/2560881130 conn(0x7f697c07a580 msgr2=0x7f697c110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:19.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 -- 172.21.15.102:0/2560881130 shutdown_connections 2024-04-26T20:34:19.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:19.995+0000 7f6982a00700 1 -- 172.21.15.102:0/2560881130 wait complete. 2024-04-26T20:34:19.998 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-26T20:34:20.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:19 smithi112 ceph-mon[32769]: pgmap v376: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:34:20.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:19 smithi130 ceph-mon[32704]: pgmap v376: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:34:20.811 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-04-26T20:34:18.562915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651850,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":973,"num_read_kb":603,"num_write":427,"num_write_kb":3073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181358,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":350964,"kb_used_data":14216,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124590860,"statfs":{"total":1151940427776,"available":1151581040640,"internally_reserved":0,"allocated":14557184,"data_stored":7941548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":344634682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":129,"num_read_kb":69,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005268"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385476+0000","last_change":"2024-04-26T20:32:26.249504+0000","last_active":"2024-04-26T20:33:30.385476+0000","last_peered":"2024-04-26T20:33:30.385476+0000","last_clean":"2024-04-26T20:33:30.385476+0000","last_became_active":"2024-04-26T20:32:26.249188+0000","last_became_peered":"2024-04-26T20:32:26.249188+0000","last_unstale":"2024-04-26T20:33:30.385476+0000","last_undegraded":"2024-04-26T20:33:30.385476+0000","last_fullsized":"2024-04-26T20:33:30.385476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:28:15.906199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394564+0000","last_change":"2024-04-26T20:32:26.246991+0000","last_active":"2024-04-26T20:33:30.394564+0000","last_peered":"2024-04-26T20:33:30.394564+0000","last_clean":"2024-04-26T20:33:30.394564+0000","last_became_active":"2024-04-26T20:32:26.245552+0000","last_became_peered":"2024-04-26T20:32:26.245552+0000","last_unstale":"2024-04-26T20:33:30.394564+0000","last_undegraded":"2024-04-26T20:33:30.394564+0000","last_fullsized":"2024-04-26T20:33:30.394564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:59:46.487659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394679+0000","last_change":"2024-04-26T20:32:26.241451+0000","last_active":"2024-04-26T20:33:30.394679+0000","last_peered":"2024-04-26T20:33:30.394679+0000","last_clean":"2024-04-26T20:33:30.394679+0000","last_became_active":"2024-04-26T20:32:26.240012+0000","last_became_peered":"2024-04-26T20:32:26.240012+0000","last_unstale":"2024-04-26T20:33:30.394679+0000","last_undegraded":"2024-04-26T20:33:30.394679+0000","last_fullsized":"2024-04-26T20:33:30.394679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:43:12.776082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384131+0000","last_change":"2024-04-26T20:32:26.224138+0000","last_active":"2024-04-26T20:33:30.384131+0000","last_peered":"2024-04-26T20:33:30.384131+0000","last_clean":"2024-04-26T20:33:30.384131+0000","last_became_active":"2024-04-26T20:32:26.223827+0000","last_became_peered":"2024-04-26T20:32:26.223827+0000","last_unstale":"2024-04-26T20:33:30.384131+0000","last_undegraded":"2024-04-26T20:33:30.384131+0000","last_fullsized":"2024-04-26T20:33:30.384131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:38:33.323001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382762+0000","last_change":"2024-04-26T20:32:26.240298+0000","last_active":"2024-04-26T20:33:30.382762+0000","last_peered":"2024-04-26T20:33:30.382762+0000","last_clean":"2024-04-26T20:33:30.382762+0000","last_became_active":"2024-04-26T20:32:26.239933+0000","last_became_peered":"2024-04-26T20:32:26.239933+0000","last_unstale":"2024-04-26T20:33:30.382762+0000","last_undegraded":"2024-04-26T20:33:30.382762+0000","last_fullsized":"2024-04-26T20:33:30.382762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:00.618136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.406074+0000","last_change":"2024-04-26T20:33:30.406074+0000","last_active":"2024-04-26T20:33:30.406074+0000","last_peered":"2024-04-26T20:33:30.406074+0000","last_clean":"2024-04-26T20:33:30.406074+0000","last_became_active":"2024-04-26T20:33:30.405818+0000","last_became_peered":"2024-04-26T20:33:30.405818+0000","last_unstale":"2024-04-26T20:33:30.406074+0000","last_undegraded":"2024-04-26T20:33:30.406074+0000","last_fullsized":"2024-04-26T20:33:30.406074+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:15:38.754348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395078+0000","last_change":"2024-04-26T20:32:26.232600+0000","last_active":"2024-04-26T20:33:30.395078+0000","last_peered":"2024-04-26T20:33:30.395078+0000","last_clean":"2024-04-26T20:33:30.395078+0000","last_became_active":"2024-04-26T20:32:26.232320+0000","last_became_peered":"2024-04-26T20:32:26.232320+0000","last_unstale":"2024-04-26T20:33:30.395078+0000","last_undegraded":"2024-04-26T20:33:30.395078+0000","last_fullsized":"2024-04-26T20:33:30.395078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:18.199940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.405729+0000","last_change":"2024-04-26T20:33:30.405729+0000","last_active":"2024-04-26T20:33:30.405729+0000","last_peered":"2024-04-26T20:33:30.405729+0000","last_clean":"2024-04-26T20:33:30.405729+0000","last_became_active":"2024-04-26T20:33:30.404668+0000","last_became_peered":"2024-04-26T20:33:30.404668+0000","last_unstale":"2024-04-26T20:33:30.405729+0000","last_undegraded":"2024-04-26T20:33:30.405729+0000","last_fullsized":"2024-04-26T20:33:30.405729+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:21:59.224049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.411139+0000","last_change":"2024-04-26T20:32:26.230224+0000","last_active":"2024-04-26T20:33:30.411139+0000","last_peered":"2024-04-26T20:33:30.411139+0000","last_clean":"2024-04-26T20:33:30.411139+0000","last_became_active":"2024-04-26T20:32:26.229865+0000","last_became_peered":"2024-04-26T20:32:26.229865+0000","last_unstale":"2024-04-26T20:33:30.411139+0000","last_undegraded":"2024-04-26T20:33:30.411139+0000","last_fullsized":"2024-04-26T20:33:30.411139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:03.060925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.386331+0000","last_change":"2024-04-26T20:32:26.219841+0000","last_active":"2024-04-26T20:33:30.386331+0000","last_peered":"2024-04-26T20:33:30.386331+0000","last_clean":"2024-04-26T20:33:30.386331+0000","last_became_active":"2024-04-26T20:32:26.219505+0000","last_became_peered":"2024-04-26T20:32:26.219505+0000","last_unstale":"2024-04-26T20:33:30.386331+0000","last_undegraded":"2024-04-26T20:33:30.386331+0000","last_fullsized":"2024-04-26T20:33:30.386331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:11.514510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395570+0000","last_change":"2024-04-26T20:32:26.226427+0000","last_active":"2024-04-26T20:33:30.395570+0000","last_peered":"2024-04-26T20:33:30.395570+0000","last_clean":"2024-04-26T20:33:30.395570+0000","last_became_active":"2024-04-26T20:32:26.226112+0000","last_became_peered":"2024-04-26T20:32:26.226112+0000","last_unstale":"2024-04-26T20:33:30.395570+0000","last_undegraded":"2024-04-26T20:33:30.395570+0000","last_fullsized":"2024-04-26T20:33:30.395570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:56.506369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384019+0000","last_change":"2024-04-26T20:32:26.222756+0000","last_active":"2024-04-26T20:33:30.384019+0000","last_peered":"2024-04-26T20:33:30.384019+0000","last_clean":"2024-04-26T20:33:30.384019+0000","last_became_active":"2024-04-26T20:32:26.222342+0000","last_became_peered":"2024-04-26T20:32:26.222342+0000","last_unstale":"2024-04-26T20:33:30.384019+0000","last_undegraded":"2024-04-26T20:33:30.384019+0000","last_fullsized":"2024-04-26T20:33:30.384019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:11.543434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.407864+0000","last_change":"2024-04-26T20:33:30.407864+0000","last_active":"2024-04-26T20:33:30.407864+0000","last_peered":"2024-04-26T20:33:30.407864+0000","last_clean":"2024-04-26T20:33:30.407864+0000","last_became_active":"2024-04-26T20:33:30.407252+0000","last_became_peered":"2024-04-26T20:33:30.407252+0000","last_unstale":"2024-04-26T20:33:30.407864+0000","last_undegraded":"2024-04-26T20:33:30.407864+0000","last_fullsized":"2024-04-26T20:33:30.407864+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:28.664781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393728+0000","last_change":"2024-04-26T20:32:26.247783+0000","last_active":"2024-04-26T20:33:30.393728+0000","last_peered":"2024-04-26T20:33:30.393728+0000","last_clean":"2024-04-26T20:33:30.393728+0000","last_became_active":"2024-04-26T20:32:26.247466+0000","last_became_peered":"2024-04-26T20:32:26.247466+0000","last_unstale":"2024-04-26T20:33:30.393728+0000","last_undegraded":"2024-04-26T20:33:30.393728+0000","last_fullsized":"2024-04-26T20:33:30.393728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:36:59.424377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393596+0000","last_change":"2024-04-26T20:32:26.249823+0000","last_active":"2024-04-26T20:33:30.393596+0000","last_peered":"2024-04-26T20:33:30.393596+0000","last_clean":"2024-04-26T20:33:30.393596+0000","last_became_active":"2024-04-26T20:32:26.248954+0000","last_became_peered":"2024-04-26T20:32:26.248954+0000","last_unstale":"2024-04-26T20:33:30.393596+0000","last_undegraded":"2024-04-26T20:33:30.393596+0000","last_fullsized":"2024-04-26T20:33:30.393596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:55:44.246537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410577+0000","last_change":"2024-04-26T20:32:26.241997+0000","last_active":"2024-04-26T20:33:30.410577+0000","last_peered":"2024-04-26T20:33:30.410577+0000","last_clean":"2024-04-26T20:33:30.410577+0000","last_became_active":"2024-04-26T20:32:26.240856+0000","last_became_peered":"2024-04-26T20:32:26.240856+0000","last_unstale":"2024-04-26T20:33:30.410577+0000","last_undegraded":"2024-04-26T20:33:30.410577+0000","last_fullsized":"2024-04-26T20:33:30.410577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:53.895994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385301+0000","last_change":"2024-04-26T20:32:26.241097+0000","last_active":"2024-04-26T20:33:30.385301+0000","last_peered":"2024-04-26T20:33:30.385301+0000","last_clean":"2024-04-26T20:33:30.385301+0000","last_became_active":"2024-04-26T20:32:26.240770+0000","last_became_peered":"2024-04-26T20:32:26.240770+0000","last_unstale":"2024-04-26T20:33:30.385301+0000","last_undegraded":"2024-04-26T20:33:30.385301+0000","last_fullsized":"2024-04-26T20:33:30.385301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:10.443314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.402985+0000","last_change":"2024-04-26T20:32:26.232279+0000","last_active":"2024-04-26T20:33:30.402985+0000","last_peered":"2024-04-26T20:33:30.402985+0000","last_clean":"2024-04-26T20:33:30.402985+0000","last_became_active":"2024-04-26T20:32:26.231256+0000","last_became_peered":"2024-04-26T20:32:26.231256+0000","last_unstale":"2024-04-26T20:33:30.402985+0000","last_undegraded":"2024-04-26T20:33:30.402985+0000","last_fullsized":"2024-04-26T20:33:30.402985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:26.484932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393047+0000","last_change":"2024-04-26T20:32:26.249876+0000","last_active":"2024-04-26T20:33:30.393047+0000","last_peered":"2024-04-26T20:33:30.393047+0000","last_clean":"2024-04-26T20:33:30.393047+0000","last_became_active":"2024-04-26T20:32:26.249416+0000","last_became_peered":"2024-04-26T20:32:26.249416+0000","last_unstale":"2024-04-26T20:33:30.393047+0000","last_undegraded":"2024-04-26T20:33:30.393047+0000","last_fullsized":"2024-04-26T20:33:30.393047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:54.616941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391854+0000","last_change":"2024-04-26T20:32:30.389128+0000","last_active":"2024-04-26T20:33:30.391854+0000","last_peered":"2024-04-26T20:33:30.391854+0000","last_clean":"2024-04-26T20:33:30.391854+0000","last_became_active":"2024-04-26T20:32:30.388668+0000","last_became_peered":"2024-04-26T20:32:30.388668+0000","last_unstale":"2024-04-26T20:33:30.391854+0000","last_undegraded":"2024-04-26T20:33:30.391854+0000","last_fullsized":"2024-04-26T20:33:30.391854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:12.019691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:13.878019+0000","last_change":"2024-04-26T20:32:25.278417+0000","last_active":"2024-04-26T20:34:13.878019+0000","last_peered":"2024-04-26T20:34:13.878019+0000","last_clean":"2024-04-26T20:34:13.878019+0000","last_became_active":"2024-04-26T20:32:25.278004+0000","last_became_peered":"2024-04-26T20:32:25.278004+0000","last_unstale":"2024-04-26T20:34:13.878019+0000","last_undegraded":"2024-04-26T20:34:13.878019+0000","last_fullsized":"2024-04-26T20:34:13.878019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:51.458285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.387891+0000","last_change":"2024-04-26T20:32:26.229015+0000","last_active":"2024-04-26T20:33:30.387891+0000","last_peered":"2024-04-26T20:33:30.387891+0000","last_clean":"2024-04-26T20:33:30.387891+0000","last_became_active":"2024-04-26T20:32:26.228729+0000","last_became_peered":"2024-04-26T20:32:26.228729+0000","last_unstale":"2024-04-26T20:33:30.387891+0000","last_undegraded":"2024-04-26T20:33:30.387891+0000","last_fullsized":"2024-04-26T20:33:30.387891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:20.014133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382866+0000","last_change":"2024-04-26T20:32:30.383196+0000","last_active":"2024-04-26T20:33:30.382866+0000","last_peered":"2024-04-26T20:33:30.382866+0000","last_clean":"2024-04-26T20:33:30.382866+0000","last_became_active":"2024-04-26T20:32:30.382831+0000","last_became_peered":"2024-04-26T20:32:30.382831+0000","last_unstale":"2024-04-26T20:33:30.382866+0000","last_undegraded":"2024-04-26T20:33:30.382866+0000","last_fullsized":"2024-04-26T20:33:30.382866+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:48:13.696394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.390035+0000","last_change":"2024-04-26T20:32:25.294227+0000","last_active":"2024-04-26T20:33:30.390035+0000","last_peered":"2024-04-26T20:33:30.390035+0000","last_clean":"2024-04-26T20:33:30.390035+0000","last_became_active":"2024-04-26T20:32:25.287681+0000","last_became_peered":"2024-04-26T20:32:25.287681+0000","last_unstale":"2024-04-26T20:33:30.390035+0000","last_undegraded":"2024-04-26T20:33:30.390035+0000","last_fullsized":"2024-04-26T20:33:30.390035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:25.567135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395225+0000","last_change":"2024-04-26T20:32:26.251364+0000","last_active":"2024-04-26T20:33:30.395225+0000","last_peered":"2024-04-26T20:33:30.395225+0000","last_clean":"2024-04-26T20:33:30.395225+0000","last_became_active":"2024-04-26T20:32:26.251037+0000","last_became_peered":"2024-04-26T20:32:26.251037+0000","last_unstale":"2024-04-26T20:33:30.395225+0000","last_undegraded":"2024-04-26T20:33:30.395225+0000","last_fullsized":"2024-04-26T20:33:30.395225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:17:41.813889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394688+0000","last_change":"2024-04-26T20:33:30.394688+0000","last_active":"2024-04-26T20:33:30.394688+0000","last_peered":"2024-04-26T20:33:30.394688+0000","last_clean":"2024-04-26T20:33:30.394688+0000","last_became_active":"2024-04-26T20:33:30.394199+0000","last_became_peered":"2024-04-26T20:33:30.394199+0000","last_unstale":"2024-04-26T20:33:30.394688+0000","last_undegraded":"2024-04-26T20:33:30.394688+0000","last_fullsized":"2024-04-26T20:33:30.394688+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:09:25.418017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383314+0000","last_change":"2024-04-26T20:32:25.267156+0000","last_active":"2024-04-26T20:33:30.383314+0000","last_peered":"2024-04-26T20:33:30.383314+0000","last_clean":"2024-04-26T20:33:30.383314+0000","last_became_active":"2024-04-26T20:32:25.266113+0000","last_became_peered":"2024-04-26T20:32:25.266113+0000","last_unstale":"2024-04-26T20:33:30.383314+0000","last_undegraded":"2024-04-26T20:33:30.383314+0000","last_fullsized":"2024-04-26T20:33:30.383314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:30:38.798437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410697+0000","last_change":"2024-04-26T20:32:26.249908+0000","last_active":"2024-04-26T20:33:30.410697+0000","last_peered":"2024-04-26T20:33:30.410697+0000","last_clean":"2024-04-26T20:33:30.410697+0000","last_became_active":"2024-04-26T20:32:26.249576+0000","last_became_peered":"2024-04-26T20:32:26.249576+0000","last_unstale":"2024-04-26T20:33:30.410697+0000","last_undegraded":"2024-04-26T20:33:30.410697+0000","last_fullsized":"2024-04-26T20:33:30.410697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:37:09.611624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382867+0000","last_change":"2024-04-26T20:32:30.391456+0000","last_active":"2024-04-26T20:33:30.382867+0000","last_peered":"2024-04-26T20:33:30.382867+0000","last_clean":"2024-04-26T20:33:30.382867+0000","last_became_active":"2024-04-26T20:32:30.391028+0000","last_became_peered":"2024-04-26T20:32:30.391028+0000","last_unstale":"2024-04-26T20:33:30.382867+0000","last_undegraded":"2024-04-26T20:33:30.382867+0000","last_fullsized":"2024-04-26T20:33:30.382867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:55:17.987421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383050+0000","last_change":"2024-04-26T20:32:25.264519+0000","last_active":"2024-04-26T20:33:30.383050+0000","last_peered":"2024-04-26T20:33:30.383050+0000","last_clean":"2024-04-26T20:33:30.383050+0000","last_became_active":"2024-04-26T20:32:25.264189+0000","last_became_peered":"2024-04-26T20:32:25.264189+0000","last_unstale":"2024-04-26T20:33:30.383050+0000","last_undegraded":"2024-04-26T20:33:30.383050+0000","last_fullsized":"2024-04-26T20:33:30.383050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:12:56.167119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391829+0000","last_change":"2024-04-26T20:32:26.226281+0000","last_active":"2024-04-26T20:33:30.391829+0000","last_peered":"2024-04-26T20:33:30.391829+0000","last_clean":"2024-04-26T20:33:30.391829+0000","last_became_active":"2024-04-26T20:32:26.225958+0000","last_became_peered":"2024-04-26T20:32:26.225958+0000","last_unstale":"2024-04-26T20:33:30.391829+0000","last_undegraded":"2024-04-26T20:33:30.391829+0000","last_fullsized":"2024-04-26T20:33:30.391829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:22:30.914293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391769+0000","last_change":"2024-04-26T20:32:30.379734+0000","last_active":"2024-04-26T20:33:30.391769+0000","last_peered":"2024-04-26T20:33:30.391769+0000","last_clean":"2024-04-26T20:33:30.391769+0000","last_became_active":"2024-04-26T20:32:30.379240+0000","last_became_peered":"2024-04-26T20:32:30.379240+0000","last_unstale":"2024-04-26T20:33:30.391769+0000","last_undegraded":"2024-04-26T20:33:30.391769+0000","last_fullsized":"2024-04-26T20:33:30.391769+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:05:46.669583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:12.454453+0000","last_change":"2024-04-26T20:33:30.400010+0000","last_active":"2024-04-26T20:34:12.454453+0000","last_peered":"2024-04-26T20:34:12.454453+0000","last_clean":"2024-04-26T20:34:12.454453+0000","last_became_active":"2024-04-26T20:33:30.396163+0000","last_became_peered":"2024-04-26T20:33:30.396163+0000","last_unstale":"2024-04-26T20:34:12.454453+0000","last_undegraded":"2024-04-26T20:34:12.454453+0000","last_fullsized":"2024-04-26T20:34:12.454453+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:43:47.524772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382753+0000","last_change":"2024-04-26T20:32:26.235743+0000","last_active":"2024-04-26T20:33:30.382753+0000","last_peered":"2024-04-26T20:33:30.382753+0000","last_clean":"2024-04-26T20:33:30.382753+0000","last_became_active":"2024-04-26T20:32:26.235375+0000","last_became_peered":"2024-04-26T20:32:26.235375+0000","last_unstale":"2024-04-26T20:33:30.382753+0000","last_undegraded":"2024-04-26T20:33:30.382753+0000","last_fullsized":"2024-04-26T20:33:30.382753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:27.673837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.403841+0000","last_change":"2024-04-26T20:33:30.403841+0000","last_active":"2024-04-26T20:33:30.403841+0000","last_peered":"2024-04-26T20:33:30.403841+0000","last_clean":"2024-04-26T20:33:30.403841+0000","last_became_active":"2024-04-26T20:33:30.403355+0000","last_became_peered":"2024-04-26T20:33:30.403355+0000","last_unstale":"2024-04-26T20:33:30.403841+0000","last_undegraded":"2024-04-26T20:33:30.403841+0000","last_fullsized":"2024-04-26T20:33:30.403841+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:27.784185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385778+0000","last_change":"2024-04-26T20:32:25.301063+0000","last_active":"2024-04-26T20:33:30.385778+0000","last_peered":"2024-04-26T20:33:30.385778+0000","last_clean":"2024-04-26T20:33:30.385778+0000","last_became_active":"2024-04-26T20:32:25.300755+0000","last_became_peered":"2024-04-26T20:32:25.300755+0000","last_unstale":"2024-04-26T20:33:30.385778+0000","last_undegraded":"2024-04-26T20:33:30.385778+0000","last_fullsized":"2024-04-26T20:33:30.385778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:50.542961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393168+0000","last_change":"2024-04-26T20:32:26.223507+0000","last_active":"2024-04-26T20:33:30.393168+0000","last_peered":"2024-04-26T20:33:30.393168+0000","last_clean":"2024-04-26T20:33:30.393168+0000","last_became_active":"2024-04-26T20:32:26.222991+0000","last_became_peered":"2024-04-26T20:32:26.222991+0000","last_unstale":"2024-04-26T20:33:30.393168+0000","last_undegraded":"2024-04-26T20:33:30.393168+0000","last_fullsized":"2024-04-26T20:33:30.393168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:53:25.207884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385297+0000","last_change":"2024-04-26T20:32:30.389354+0000","last_active":"2024-04-26T20:33:30.385297+0000","last_peered":"2024-04-26T20:33:30.385297+0000","last_clean":"2024-04-26T20:33:30.385297+0000","last_became_active":"2024-04-26T20:32:30.388848+0000","last_became_peered":"2024-04-26T20:32:30.388848+0000","last_unstale":"2024-04-26T20:33:30.385297+0000","last_undegraded":"2024-04-26T20:33:30.385297+0000","last_fullsized":"2024-04-26T20:33:30.385297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:12.615750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.399639+0000","last_change":"2024-04-26T20:32:25.292694+0000","last_active":"2024-04-26T20:33:30.399639+0000","last_peered":"2024-04-26T20:33:30.399639+0000","last_clean":"2024-04-26T20:33:30.399639+0000","last_became_active":"2024-04-26T20:32:25.283757+0000","last_became_peered":"2024-04-26T20:32:25.283757+0000","last_unstale":"2024-04-26T20:33:30.399639+0000","last_undegraded":"2024-04-26T20:33:30.399639+0000","last_fullsized":"2024-04-26T20:33:30.399639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:05:15.294842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384573+0000","last_change":"2024-04-26T20:32:26.232770+0000","last_active":"2024-04-26T20:33:30.384573+0000","last_peered":"2024-04-26T20:33:30.384573+0000","last_clean":"2024-04-26T20:33:30.384573+0000","last_became_active":"2024-04-26T20:32:26.232381+0000","last_became_peered":"2024-04-26T20:32:26.232381+0000","last_unstale":"2024-04-26T20:33:30.384573+0000","last_undegraded":"2024-04-26T20:33:30.384573+0000","last_fullsized":"2024-04-26T20:33:30.384573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:16:43.310582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394108+0000","last_change":"2024-04-26T20:32:30.389690+0000","last_active":"2024-04-26T20:33:30.394108+0000","last_peered":"2024-04-26T20:33:30.394108+0000","last_clean":"2024-04-26T20:33:30.394108+0000","last_became_active":"2024-04-26T20:32:30.389258+0000","last_became_peered":"2024-04-26T20:32:30.389258+0000","last_unstale":"2024-04-26T20:33:30.394108+0000","last_undegraded":"2024-04-26T20:33:30.394108+0000","last_fullsized":"2024-04-26T20:33:30.394108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:36.528757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391527+0000","last_change":"2024-04-26T20:32:25.267681+0000","last_active":"2024-04-26T20:33:30.391527+0000","last_peered":"2024-04-26T20:33:30.391527+0000","last_clean":"2024-04-26T20:33:30.391527+0000","last_became_active":"2024-04-26T20:32:25.267351+0000","last_became_peered":"2024-04-26T20:32:25.267351+0000","last_unstale":"2024-04-26T20:33:30.391527+0000","last_undegraded":"2024-04-26T20:33:30.391527+0000","last_fullsized":"2024-04-26T20:33:30.391527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:00:26.339458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393346+0000","last_change":"2024-04-26T20:32:26.243638+0000","last_active":"2024-04-26T20:33:30.393346+0000","last_peered":"2024-04-26T20:33:30.393346+0000","last_clean":"2024-04-26T20:33:30.393346+0000","last_became_active":"2024-04-26T20:32:26.241783+0000","last_became_peered":"2024-04-26T20:32:26.241783+0000","last_unstale":"2024-04-26T20:33:30.393346+0000","last_undegraded":"2024-04-26T20:33:30.393346+0000","last_fullsized":"2024-04-26T20:33:30.393346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:20.821967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384804+0000","last_change":"2024-04-26T20:32:30.382937+0000","last_active":"2024-04-26T20:33:30.384804+0000","last_peered":"2024-04-26T20:33:30.384804+0000","last_clean":"2024-04-26T20:33:30.384804+0000","last_became_active":"2024-04-26T20:32:30.378828+0000","last_became_peered":"2024-04-26T20:32:30.378828+0000","last_unstale":"2024-04-26T20:33:30.384804+0000","last_undegraded":"2024-04-26T20:33:30.384804+0000","last_fullsized":"2024-04-26T20:33:30.384804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:33:41.685012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":195,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:16.830304+0000","last_change":"2024-04-26T20:32:25.280362+0000","last_active":"2024-04-26T20:34:16.830304+0000","last_peered":"2024-04-26T20:34:16.830304+0000","last_clean":"2024-04-26T20:34:16.830304+0000","last_became_active":"2024-04-26T20:32:25.279975+0000","last_became_peered":"2024-04-26T20:32:25.279975+0000","last_unstale":"2024-04-26T20:34:16.830304+0000","last_undegraded":"2024-04-26T20:34:16.830304+0000","last_fullsized":"2024-04-26T20:34:16.830304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:08.150635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":164,"num_read_kb":164,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.411093+0000","last_change":"2024-04-26T20:32:26.231231+0000","last_active":"2024-04-26T20:33:30.411093+0000","last_peered":"2024-04-26T20:33:30.411093+0000","last_clean":"2024-04-26T20:33:30.411093+0000","last_became_active":"2024-04-26T20:32:26.230831+0000","last_became_peered":"2024-04-26T20:32:26.230831+0000","last_unstale":"2024-04-26T20:33:30.411093+0000","last_undegraded":"2024-04-26T20:33:30.411093+0000","last_fullsized":"2024-04-26T20:33:30.411093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:03:45.272542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.401129+0000","last_change":"2024-04-26T20:32:30.382633+0000","last_active":"2024-04-26T20:33:30.401129+0000","last_peered":"2024-04-26T20:33:30.401129+0000","last_clean":"2024-04-26T20:33:30.401129+0000","last_became_active":"2024-04-26T20:32:30.379821+0000","last_became_peered":"2024-04-26T20:32:30.379821+0000","last_unstale":"2024-04-26T20:33:30.401129+0000","last_undegraded":"2024-04-26T20:33:30.401129+0000","last_fullsized":"2024-04-26T20:33:30.401129+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:58:16.888172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395693+0000","last_change":"2024-04-26T20:32:25.264844+0000","last_active":"2024-04-26T20:33:30.395693+0000","last_peered":"2024-04-26T20:33:30.395693+0000","last_clean":"2024-04-26T20:33:30.395693+0000","last_became_active":"2024-04-26T20:32:25.264097+0000","last_became_peered":"2024-04-26T20:32:25.264097+0000","last_unstale":"2024-04-26T20:33:30.395693+0000","last_undegraded":"2024-04-26T20:33:30.395693+0000","last_fullsized":"2024-04-26T20:33:30.395693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:09.308656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.400892+0000","last_change":"2024-04-26T20:33:30.400892+0000","last_active":"2024-04-26T20:33:30.400892+0000","last_peered":"2024-04-26T20:33:30.400892+0000","last_clean":"2024-04-26T20:33:30.400892+0000","last_became_active":"2024-04-26T20:33:30.398674+0000","last_became_peered":"2024-04-26T20:33:30.398674+0000","last_unstale":"2024-04-26T20:33:30.400892+0000","last_undegraded":"2024-04-26T20:33:30.400892+0000","last_fullsized":"2024-04-26T20:33:30.400892+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:03:08.573433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.412074+0000","last_change":"2024-04-26T20:33:30.412074+0000","last_active":"2024-04-26T20:33:30.412074+0000","last_peered":"2024-04-26T20:33:30.412074+0000","last_clean":"2024-04-26T20:33:30.412074+0000","last_became_active":"2024-04-26T20:33:30.411535+0000","last_became_peered":"2024-04-26T20:33:30.411535+0000","last_unstale":"2024-04-26T20:33:30.412074+0000","last_undegraded":"2024-04-26T20:33:30.412074+0000","last_fullsized":"2024-04-26T20:33:30.412074+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:54:52.809109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,10],"acting":[4,1,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391704+0000","last_change":"2024-04-26T20:32:25.284996+0000","last_active":"2024-04-26T20:33:30.391704+0000","last_peered":"2024-04-26T20:33:30.391704+0000","last_clean":"2024-04-26T20:33:30.391704+0000","last_became_active":"2024-04-26T20:32:25.284702+0000","last_became_peered":"2024-04-26T20:32:25.284702+0000","last_unstale":"2024-04-26T20:33:30.391704+0000","last_undegraded":"2024-04-26T20:33:30.391704+0000","last_fullsized":"2024-04-26T20:33:30.391704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:39.831522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410826+0000","last_change":"2024-04-26T20:32:26.230357+0000","last_active":"2024-04-26T20:33:30.410826+0000","last_peered":"2024-04-26T20:33:30.410826+0000","last_clean":"2024-04-26T20:33:30.410826+0000","last_became_active":"2024-04-26T20:32:26.229831+0000","last_became_peered":"2024-04-26T20:32:26.229831+0000","last_unstale":"2024-04-26T20:33:30.410826+0000","last_undegraded":"2024-04-26T20:33:30.410826+0000","last_fullsized":"2024-04-26T20:33:30.410826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:20:12.452768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.401654+0000","last_change":"2024-04-26T20:32:30.383469+0000","last_active":"2024-04-26T20:33:30.401654+0000","last_peered":"2024-04-26T20:33:30.401654+0000","last_clean":"2024-04-26T20:33:30.401654+0000","last_became_active":"2024-04-26T20:32:30.382416+0000","last_became_peered":"2024-04-26T20:32:30.382416+0000","last_unstale":"2024-04-26T20:33:30.401654+0000","last_undegraded":"2024-04-26T20:33:30.401654+0000","last_fullsized":"2024-04-26T20:33:30.401654+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:55.254214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":231,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:17.460831+0000","last_change":"2024-04-26T20:32:25.264433+0000","last_active":"2024-04-26T20:34:17.460831+0000","last_peered":"2024-04-26T20:34:17.460831+0000","last_clean":"2024-04-26T20:34:17.460831+0000","last_became_active":"2024-04-26T20:32:25.264104+0000","last_became_peered":"2024-04-26T20:32:25.264104+0000","last_unstale":"2024-04-26T20:34:17.460831+0000","last_undegraded":"2024-04-26T20:34:17.460831+0000","last_fullsized":"2024-04-26T20:34:17.460831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.030082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54826,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":169,"num_read_kb":137,"num_write":37,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383114+0000","last_change":"2024-04-26T20:32:26.219704+0000","last_active":"2024-04-26T20:33:30.383114+0000","last_peered":"2024-04-26T20:33:30.383114+0000","last_clean":"2024-04-26T20:33:30.383114+0000","last_became_active":"2024-04-26T20:32:26.219397+0000","last_became_peered":"2024-04-26T20:32:26.219397+0000","last_unstale":"2024-04-26T20:33:30.383114+0000","last_undegraded":"2024-04-26T20:33:30.383114+0000","last_fullsized":"2024-04-26T20:33:30.383114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:04.570044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.392486+0000","last_change":"2024-04-26T20:32:30.388828+0000","last_active":"2024-04-26T20:33:30.392486+0000","last_peered":"2024-04-26T20:33:30.392486+0000","last_clean":"2024-04-26T20:33:30.392486+0000","last_became_active":"2024-04-26T20:32:30.387748+0000","last_became_peered":"2024-04-26T20:32:30.387748+0000","last_unstale":"2024-04-26T20:33:30.392486+0000","last_undegraded":"2024-04-26T20:33:30.392486+0000","last_fullsized":"2024-04-26T20:33:30.392486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:53:09.900151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391784+0000","last_change":"2024-04-26T20:32:25.271910+0000","last_active":"2024-04-26T20:33:30.391784+0000","last_peered":"2024-04-26T20:33:30.391784+0000","last_clean":"2024-04-26T20:33:30.391784+0000","last_became_active":"2024-04-26T20:32:25.271572+0000","last_became_peered":"2024-04-26T20:32:25.271572+0000","last_unstale":"2024-04-26T20:33:30.391784+0000","last_undegraded":"2024-04-26T20:33:30.391784+0000","last_fullsized":"2024-04-26T20:33:30.391784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:32:59.013229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395265+0000","last_change":"2024-04-26T20:32:26.237691+0000","last_active":"2024-04-26T20:33:30.395265+0000","last_peered":"2024-04-26T20:33:30.395265+0000","last_clean":"2024-04-26T20:33:30.395265+0000","last_became_active":"2024-04-26T20:32:26.237338+0000","last_became_peered":"2024-04-26T20:32:26.237338+0000","last_unstale":"2024-04-26T20:33:30.395265+0000","last_undegraded":"2024-04-26T20:33:30.395265+0000","last_fullsized":"2024-04-26T20:33:30.395265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:51.782689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385493+0000","last_change":"2024-04-26T20:32:30.388485+0000","last_active":"2024-04-26T20:33:30.385493+0000","last_peered":"2024-04-26T20:33:30.385493+0000","last_clean":"2024-04-26T20:33:30.385493+0000","last_became_active":"2024-04-26T20:32:30.388074+0000","last_became_peered":"2024-04-26T20:32:30.388074+0000","last_unstale":"2024-04-26T20:33:30.385493+0000","last_undegraded":"2024-04-26T20:33:30.385493+0000","last_fullsized":"2024-04-26T20:33:30.385493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:52.792551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391873+0000","last_change":"2024-04-26T20:32:25.271589+0000","last_active":"2024-04-26T20:33:30.391873+0000","last_peered":"2024-04-26T20:33:30.391873+0000","last_clean":"2024-04-26T20:33:30.391873+0000","last_became_active":"2024-04-26T20:32:25.271260+0000","last_became_peered":"2024-04-26T20:32:25.271260+0000","last_unstale":"2024-04-26T20:33:30.391873+0000","last_undegraded":"2024-04-26T20:33:30.391873+0000","last_fullsized":"2024-04-26T20:33:30.391873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:30.978561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395338+0000","last_change":"2024-04-26T20:32:26.240158+0000","last_active":"2024-04-26T20:33:30.395338+0000","last_peered":"2024-04-26T20:33:30.395338+0000","last_clean":"2024-04-26T20:33:30.395338+0000","last_became_active":"2024-04-26T20:32:26.239831+0000","last_became_peered":"2024-04-26T20:32:26.239831+0000","last_unstale":"2024-04-26T20:33:30.395338+0000","last_undegraded":"2024-04-26T20:33:30.395338+0000","last_fullsized":"2024-04-26T20:33:30.395338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:07.870097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.401518+0000","last_change":"2024-04-26T20:32:30.392090+0000","last_active":"2024-04-26T20:33:30.401518+0000","last_peered":"2024-04-26T20:33:30.401518+0000","last_clean":"2024-04-26T20:33:30.401518+0000","last_became_active":"2024-04-26T20:32:30.390404+0000","last_became_peered":"2024-04-26T20:32:30.390404+0000","last_unstale":"2024-04-26T20:33:30.401518+0000","last_undegraded":"2024-04-26T20:33:30.401518+0000","last_fullsized":"2024-04-26T20:33:30.401518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:50:10.743133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:12.877628+0000","last_change":"2024-04-26T20:32:25.264291+0000","last_active":"2024-04-26T20:34:12.877628+0000","last_peered":"2024-04-26T20:34:12.877628+0000","last_clean":"2024-04-26T20:34:12.877628+0000","last_became_active":"2024-04-26T20:32:25.263901+0000","last_became_peered":"2024-04-26T20:32:25.263901+0000","last_unstale":"2024-04-26T20:34:12.877628+0000","last_undegraded":"2024-04-26T20:34:12.877628+0000","last_fullsized":"2024-04-26T20:34:12.877628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:23:28.851941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395208+0000","last_change":"2024-04-26T20:32:26.221472+0000","last_active":"2024-04-26T20:33:30.395208+0000","last_peered":"2024-04-26T20:33:30.395208+0000","last_clean":"2024-04-26T20:33:30.395208+0000","last_became_active":"2024-04-26T20:32:26.221117+0000","last_became_peered":"2024-04-26T20:32:26.221117+0000","last_unstale":"2024-04-26T20:33:30.395208+0000","last_undegraded":"2024-04-26T20:33:30.395208+0000","last_fullsized":"2024-04-26T20:33:30.395208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:27:01.666749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.400851+0000","last_change":"2024-04-26T20:32:30.392000+0000","last_active":"2024-04-26T20:33:30.400851+0000","last_peered":"2024-04-26T20:33:30.400851+0000","last_clean":"2024-04-26T20:33:30.400851+0000","last_became_active":"2024-04-26T20:32:30.386933+0000","last_became_peered":"2024-04-26T20:32:30.386933+0000","last_unstale":"2024-04-26T20:33:30.400851+0000","last_undegraded":"2024-04-26T20:33:30.400851+0000","last_fullsized":"2024-04-26T20:33:30.400851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:23.666439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382595+0000","last_change":"2024-04-26T20:32:25.276482+0000","last_active":"2024-04-26T20:33:30.382595+0000","last_peered":"2024-04-26T20:33:30.382595+0000","last_clean":"2024-04-26T20:33:30.382595+0000","last_became_active":"2024-04-26T20:32:25.276159+0000","last_became_peered":"2024-04-26T20:32:25.276159+0000","last_unstale":"2024-04-26T20:33:30.382595+0000","last_undegraded":"2024-04-26T20:33:30.382595+0000","last_fullsized":"2024-04-26T20:33:30.382595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:34.029524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393349+0000","last_change":"2024-04-26T20:32:26.225344+0000","last_active":"2024-04-26T20:33:30.393349+0000","last_peered":"2024-04-26T20:33:30.393349+0000","last_clean":"2024-04-26T20:33:30.393349+0000","last_became_active":"2024-04-26T20:32:26.225039+0000","last_became_peered":"2024-04-26T20:32:26.225039+0000","last_unstale":"2024-04-26T20:33:30.393349+0000","last_undegraded":"2024-04-26T20:33:30.393349+0000","last_fullsized":"2024-04-26T20:33:30.393349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:08.680765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393327+0000","last_change":"2024-04-26T20:32:30.389274+0000","last_active":"2024-04-26T20:33:30.393327+0000","last_peered":"2024-04-26T20:33:30.393327+0000","last_clean":"2024-04-26T20:33:30.393327+0000","last_became_active":"2024-04-26T20:32:30.388527+0000","last_became_peered":"2024-04-26T20:32:30.388527+0000","last_unstale":"2024-04-26T20:33:30.393327+0000","last_undegraded":"2024-04-26T20:33:30.393327+0000","last_fullsized":"2024-04-26T20:33:30.393327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:14.689223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383283+0000","last_change":"2024-04-26T20:32:25.267338+0000","last_active":"2024-04-26T20:33:30.383283+0000","last_peered":"2024-04-26T20:33:30.383283+0000","last_clean":"2024-04-26T20:33:30.383283+0000","last_became_active":"2024-04-26T20:32:25.266234+0000","last_became_peered":"2024-04-26T20:32:25.266234+0000","last_unstale":"2024-04-26T20:33:30.383283+0000","last_undegraded":"2024-04-26T20:33:30.383283+0000","last_fullsized":"2024-04-26T20:33:30.383283+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:55.603147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384931+0000","last_change":"2024-04-26T20:32:26.222686+0000","last_active":"2024-04-26T20:33:30.384931+0000","last_peered":"2024-04-26T20:33:30.384931+0000","last_clean":"2024-04-26T20:33:30.384931+0000","last_became_active":"2024-04-26T20:32:26.222364+0000","last_became_peered":"2024-04-26T20:32:26.222364+0000","last_unstale":"2024-04-26T20:33:30.384931+0000","last_undegraded":"2024-04-26T20:33:30.384931+0000","last_fullsized":"2024-04-26T20:33:30.384931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:56.170913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382708+0000","last_change":"2024-04-26T20:32:30.414367+0000","last_active":"2024-04-26T20:33:30.382708+0000","last_peered":"2024-04-26T20:33:30.382708+0000","last_clean":"2024-04-26T20:33:30.382708+0000","last_became_active":"2024-04-26T20:32:30.413889+0000","last_became_peered":"2024-04-26T20:32:30.413889+0000","last_unstale":"2024-04-26T20:33:30.382708+0000","last_undegraded":"2024-04-26T20:33:30.382708+0000","last_fullsized":"2024-04-26T20:33:30.382708+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:11.336500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.402415+0000","last_change":"2024-04-26T20:32:25.286408+0000","last_active":"2024-04-26T20:33:30.402415+0000","last_peered":"2024-04-26T20:33:30.402415+0000","last_clean":"2024-04-26T20:33:30.402415+0000","last_became_active":"2024-04-26T20:32:25.285986+0000","last_became_peered":"2024-04-26T20:32:25.285986+0000","last_unstale":"2024-04-26T20:33:30.402415+0000","last_undegraded":"2024-04-26T20:33:30.402415+0000","last_fullsized":"2024-04-26T20:33:30.402415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:47.757720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.386184+0000","last_change":"2024-04-26T20:32:26.250459+0000","last_active":"2024-04-26T20:33:30.386184+0000","last_peered":"2024-04-26T20:33:30.386184+0000","last_clean":"2024-04-26T20:33:30.386184+0000","last_became_active":"2024-04-26T20:32:26.250154+0000","last_became_peered":"2024-04-26T20:32:26.250154+0000","last_unstale":"2024-04-26T20:33:30.386184+0000","last_undegraded":"2024-04-26T20:33:30.386184+0000","last_fullsized":"2024-04-26T20:33:30.386184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:47.185067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.398483+0000","last_change":"2024-04-26T20:32:30.395258+0000","last_active":"2024-04-26T20:33:30.398483+0000","last_peered":"2024-04-26T20:33:30.398483+0000","last_clean":"2024-04-26T20:33:30.398483+0000","last_became_active":"2024-04-26T20:32:30.392452+0000","last_became_peered":"2024-04-26T20:32:30.392452+0000","last_unstale":"2024-04-26T20:33:30.398483+0000","last_undegraded":"2024-04-26T20:33:30.398483+0000","last_fullsized":"2024-04-26T20:33:30.398483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:19.883901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:16.885302+0000","last_change":"2024-04-26T20:32:25.297361+0000","last_active":"2024-04-26T20:34:16.885302+0000","last_peered":"2024-04-26T20:34:16.885302+0000","last_clean":"2024-04-26T20:34:16.885302+0000","last_became_active":"2024-04-26T20:32:25.297125+0000","last_became_peered":"2024-04-26T20:32:25.297125+0000","last_unstale":"2024-04-26T20:34:16.885302+0000","last_undegraded":"2024-04-26T20:34:16.885302+0000","last_fullsized":"2024-04-26T20:34:16.885302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:14:38.038683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385946+0000","last_change":"2024-04-26T20:32:26.249371+0000","last_active":"2024-04-26T20:33:30.385946+0000","last_peered":"2024-04-26T20:33:30.385946+0000","last_clean":"2024-04-26T20:33:30.385946+0000","last_became_active":"2024-04-26T20:32:26.249018+0000","last_became_peered":"2024-04-26T20:32:26.249018+0000","last_unstale":"2024-04-26T20:33:30.385946+0000","last_undegraded":"2024-04-26T20:33:30.385946+0000","last_fullsized":"2024-04-26T20:33:30.385946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:27.553928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.401076+0000","last_change":"2024-04-26T20:32:30.414854+0000","last_active":"2024-04-26T20:33:30.401076+0000","last_peered":"2024-04-26T20:33:30.401076+0000","last_clean":"2024-04-26T20:33:30.401076+0000","last_became_active":"2024-04-26T20:32:30.414374+0000","last_became_peered":"2024-04-26T20:32:30.414374+0000","last_unstale":"2024-04-26T20:33:30.401076+0000","last_undegraded":"2024-04-26T20:33:30.401076+0000","last_fullsized":"2024-04-26T20:33:30.401076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:32.071225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391794+0000","last_change":"2024-04-26T20:32:26.222681+0000","last_active":"2024-04-26T20:33:30.391794+0000","last_peered":"2024-04-26T20:33:30.391794+0000","last_clean":"2024-04-26T20:33:30.391794+0000","last_became_active":"2024-04-26T20:32:26.222099+0000","last_became_peered":"2024-04-26T20:32:26.222099+0000","last_unstale":"2024-04-26T20:33:30.391794+0000","last_undegraded":"2024-04-26T20:33:30.391794+0000","last_fullsized":"2024-04-26T20:33:30.391794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:35:21.958180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:16.884424+0000","last_change":"2024-04-26T20:32:25.276314+0000","last_active":"2024-04-26T20:34:16.884424+0000","last_peered":"2024-04-26T20:34:16.884424+0000","last_clean":"2024-04-26T20:34:16.884424+0000","last_became_active":"2024-04-26T20:32:25.275926+0000","last_became_peered":"2024-04-26T20:32:25.275926+0000","last_unstale":"2024-04-26T20:34:16.884424+0000","last_undegraded":"2024-04-26T20:34:16.884424+0000","last_fullsized":"2024-04-26T20:34:16.884424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:25.027777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395299+0000","last_change":"2024-04-26T20:32:30.385121+0000","last_active":"2024-04-26T20:33:30.395299+0000","last_peered":"2024-04-26T20:33:30.395299+0000","last_clean":"2024-04-26T20:33:30.395299+0000","last_became_active":"2024-04-26T20:32:30.384697+0000","last_became_peered":"2024-04-26T20:32:30.384697+0000","last_unstale":"2024-04-26T20:33:30.395299+0000","last_undegraded":"2024-04-26T20:33:30.395299+0000","last_fullsized":"2024-04-26T20:33:30.395299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:00.725130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385691+0000","last_change":"2024-04-26T20:32:26.227550+0000","last_active":"2024-04-26T20:33:30.385691+0000","last_peered":"2024-04-26T20:33:30.385691+0000","last_clean":"2024-04-26T20:33:30.385691+0000","last_became_active":"2024-04-26T20:32:26.224262+0000","last_became_peered":"2024-04-26T20:32:26.224262+0000","last_unstale":"2024-04-26T20:33:30.385691+0000","last_undegraded":"2024-04-26T20:33:30.385691+0000","last_fullsized":"2024-04-26T20:33:30.385691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:01.413870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383169+0000","last_change":"2024-04-26T20:32:25.274253+0000","last_active":"2024-04-26T20:33:30.383169+0000","last_peered":"2024-04-26T20:33:30.383169+0000","last_clean":"2024-04-26T20:33:30.383169+0000","last_became_active":"2024-04-26T20:32:25.273956+0000","last_became_peered":"2024-04-26T20:32:25.273956+0000","last_unstale":"2024-04-26T20:33:30.383169+0000","last_undegraded":"2024-04-26T20:33:30.383169+0000","last_fullsized":"2024-04-26T20:33:30.383169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:23:27.832135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":353,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395071+0000","last_change":"2024-04-26T20:31:05.644347+0000","last_active":"2024-04-26T20:33:30.395071+0000","last_peered":"2024-04-26T20:33:30.395071+0000","last_clean":"2024-04-26T20:33:30.395071+0000","last_became_active":"2024-04-26T20:31:05.643888+0000","last_became_peered":"2024-04-26T20:31:05.643888+0000","last_unstale":"2024-04-26T20:33:30.395071+0000","last_undegraded":"2024-04-26T20:33:30.395071+0000","last_fullsized":"2024-04-26T20:33:30.395071+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:27:11.501328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:27:11.501328+0000","last_clean_scrub_stamp":"2024-04-26T20:27:11.501328+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:08.802687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.406111+0000","last_change":"2024-04-26T20:33:30.406111+0000","last_active":"2024-04-26T20:33:30.406111+0000","last_peered":"2024-04-26T20:33:30.406111+0000","last_clean":"2024-04-26T20:33:30.406111+0000","last_became_active":"2024-04-26T20:33:30.405639+0000","last_became_peered":"2024-04-26T20:33:30.405639+0000","last_unstale":"2024-04-26T20:33:30.406111+0000","last_undegraded":"2024-04-26T20:33:30.406111+0000","last_fullsized":"2024-04-26T20:33:30.406111+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:15:50.668699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.396839+0000","last_change":"2024-04-26T20:32:26.239936+0000","last_active":"2024-04-26T20:33:30.396839+0000","last_peered":"2024-04-26T20:33:30.396839+0000","last_clean":"2024-04-26T20:33:30.396839+0000","last_became_active":"2024-04-26T20:32:26.239583+0000","last_became_peered":"2024-04-26T20:32:26.239583+0000","last_unstale":"2024-04-26T20:33:30.396839+0000","last_undegraded":"2024-04-26T20:33:30.396839+0000","last_fullsized":"2024-04-26T20:33:30.396839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:57.183460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395107+0000","last_change":"2024-04-26T20:32:25.267876+0000","last_active":"2024-04-26T20:33:30.395107+0000","last_peered":"2024-04-26T20:33:30.395107+0000","last_clean":"2024-04-26T20:33:30.395107+0000","last_became_active":"2024-04-26T20:32:25.267573+0000","last_became_peered":"2024-04-26T20:32:25.267573+0000","last_unstale":"2024-04-26T20:33:30.395107+0000","last_undegraded":"2024-04-26T20:33:30.395107+0000","last_fullsized":"2024-04-26T20:33:30.395107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:43:55.630426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384980+0000","last_change":"2024-04-26T20:32:30.383364+0000","last_active":"2024-04-26T20:33:30.384980+0000","last_peered":"2024-04-26T20:33:30.384980+0000","last_clean":"2024-04-26T20:33:30.384980+0000","last_became_active":"2024-04-26T20:32:30.382875+0000","last_became_peered":"2024-04-26T20:32:30.382875+0000","last_unstale":"2024-04-26T20:33:30.384980+0000","last_undegraded":"2024-04-26T20:33:30.384980+0000","last_fullsized":"2024-04-26T20:33:30.384980+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:55.112888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383125+0000","last_change":"2024-04-26T20:32:25.280508+0000","last_active":"2024-04-26T20:33:30.383125+0000","last_peered":"2024-04-26T20:33:30.383125+0000","last_clean":"2024-04-26T20:33:30.383125+0000","last_became_active":"2024-04-26T20:32:25.280265+0000","last_became_peered":"2024-04-26T20:32:25.280265+0000","last_unstale":"2024-04-26T20:33:30.383125+0000","last_undegraded":"2024-04-26T20:33:30.383125+0000","last_fullsized":"2024-04-26T20:33:30.383125+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:16:50.427659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.398818+0000","last_change":"2024-04-26T20:32:26.255230+0000","last_active":"2024-04-26T20:33:30.398818+0000","last_peered":"2024-04-26T20:33:30.398818+0000","last_clean":"2024-04-26T20:33:30.398818+0000","last_became_active":"2024-04-26T20:32:26.254099+0000","last_became_peered":"2024-04-26T20:32:26.254099+0000","last_unstale":"2024-04-26T20:33:30.398818+0000","last_undegraded":"2024-04-26T20:33:30.398818+0000","last_fullsized":"2024-04-26T20:33:30.398818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:41.895816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384464+0000","last_change":"2024-04-26T20:32:30.414688+0000","last_active":"2024-04-26T20:33:30.384464+0000","last_peered":"2024-04-26T20:33:30.384464+0000","last_clean":"2024-04-26T20:33:30.384464+0000","last_became_active":"2024-04-26T20:32:30.414052+0000","last_became_peered":"2024-04-26T20:32:30.414052+0000","last_unstale":"2024-04-26T20:33:30.384464+0000","last_undegraded":"2024-04-26T20:33:30.384464+0000","last_fullsized":"2024-04-26T20:33:30.384464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:24.943559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.403192+0000","last_change":"2024-04-26T20:32:25.298609+0000","last_active":"2024-04-26T20:33:30.403192+0000","last_peered":"2024-04-26T20:33:30.403192+0000","last_clean":"2024-04-26T20:33:30.403192+0000","last_became_active":"2024-04-26T20:32:25.298155+0000","last_became_peered":"2024-04-26T20:32:25.298155+0000","last_unstale":"2024-04-26T20:33:30.403192+0000","last_undegraded":"2024-04-26T20:33:30.403192+0000","last_fullsized":"2024-04-26T20:33:30.403192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:37.130994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.388963+0000","last_change":"2024-04-26T20:32:26.231203+0000","last_active":"2024-04-26T20:33:30.388963+0000","last_peered":"2024-04-26T20:33:30.388963+0000","last_clean":"2024-04-26T20:33:30.388963+0000","last_became_active":"2024-04-26T20:32:26.230775+0000","last_became_peered":"2024-04-26T20:32:26.230775+0000","last_unstale":"2024-04-26T20:33:30.388963+0000","last_undegraded":"2024-04-26T20:33:30.388963+0000","last_fullsized":"2024-04-26T20:33:30.388963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:40.557744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391393+0000","last_change":"2024-04-26T20:32:30.375703+0000","last_active":"2024-04-26T20:33:30.391393+0000","last_peered":"2024-04-26T20:33:30.391393+0000","last_clean":"2024-04-26T20:33:30.391393+0000","last_became_active":"2024-04-26T20:32:30.375292+0000","last_became_peered":"2024-04-26T20:32:30.375292+0000","last_unstale":"2024-04-26T20:33:30.391393+0000","last_undegraded":"2024-04-26T20:33:30.391393+0000","last_fullsized":"2024-04-26T20:33:30.391393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:39.747426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383571+0000","last_change":"2024-04-26T20:32:25.290908+0000","last_active":"2024-04-26T20:33:30.383571+0000","last_peered":"2024-04-26T20:33:30.383571+0000","last_clean":"2024-04-26T20:33:30.383571+0000","last_became_active":"2024-04-26T20:32:25.290655+0000","last_became_peered":"2024-04-26T20:32:25.290655+0000","last_unstale":"2024-04-26T20:33:30.383571+0000","last_undegraded":"2024-04-26T20:33:30.383571+0000","last_fullsized":"2024-04-26T20:33:30.383571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:28:42.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394820+0000","last_change":"2024-04-26T20:32:26.246554+0000","last_active":"2024-04-26T20:33:30.394820+0000","last_peered":"2024-04-26T20:33:30.394820+0000","last_clean":"2024-04-26T20:33:30.394820+0000","last_became_active":"2024-04-26T20:32:26.246279+0000","last_became_peered":"2024-04-26T20:32:26.246279+0000","last_unstale":"2024-04-26T20:33:30.394820+0000","last_undegraded":"2024-04-26T20:33:30.394820+0000","last_fullsized":"2024-04-26T20:33:30.394820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:02.561410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383250+0000","last_change":"2024-04-26T20:32:30.414622+0000","last_active":"2024-04-26T20:33:30.383250+0000","last_peered":"2024-04-26T20:33:30.383250+0000","last_clean":"2024-04-26T20:33:30.383250+0000","last_became_active":"2024-04-26T20:32:30.414176+0000","last_became_peered":"2024-04-26T20:32:30.414176+0000","last_unstale":"2024-04-26T20:33:30.383250+0000","last_undegraded":"2024-04-26T20:33:30.383250+0000","last_fullsized":"2024-04-26T20:33:30.383250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:15.011512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410491+0000","last_change":"2024-04-26T20:32:25.275220+0000","last_active":"2024-04-26T20:33:30.410491+0000","last_peered":"2024-04-26T20:33:30.410491+0000","last_clean":"2024-04-26T20:33:30.410491+0000","last_became_active":"2024-04-26T20:32:25.274990+0000","last_became_peered":"2024-04-26T20:32:25.274990+0000","last_unstale":"2024-04-26T20:33:30.410491+0000","last_undegraded":"2024-04-26T20:33:30.410491+0000","last_fullsized":"2024-04-26T20:33:30.410491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:58.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382593+0000","last_change":"2024-04-26T20:32:26.218481+0000","last_active":"2024-04-26T20:33:30.382593+0000","last_peered":"2024-04-26T20:33:30.382593+0000","last_clean":"2024-04-26T20:33:30.382593+0000","last_became_active":"2024-04-26T20:32:26.218146+0000","last_became_peered":"2024-04-26T20:32:26.218146+0000","last_unstale":"2024-04-26T20:33:30.382593+0000","last_undegraded":"2024-04-26T20:33:30.382593+0000","last_fullsized":"2024-04-26T20:33:30.382593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:37.310945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.401948+0000","last_change":"2024-04-26T20:32:30.383815+0000","last_active":"2024-04-26T20:33:30.401948+0000","last_peered":"2024-04-26T20:33:30.401948+0000","last_clean":"2024-04-26T20:33:30.401948+0000","last_became_active":"2024-04-26T20:32:30.382818+0000","last_became_peered":"2024-04-26T20:32:30.382818+0000","last_unstale":"2024-04-26T20:33:30.401948+0000","last_undegraded":"2024-04-26T20:33:30.401948+0000","last_fullsized":"2024-04-26T20:33:30.401948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:47.839781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385330+0000","last_change":"2024-04-26T20:32:25.291527+0000","last_active":"2024-04-26T20:33:30.385330+0000","last_peered":"2024-04-26T20:33:30.385330+0000","last_clean":"2024-04-26T20:33:30.385330+0000","last_became_active":"2024-04-26T20:32:25.285353+0000","last_became_peered":"2024-04-26T20:32:25.285353+0000","last_unstale":"2024-04-26T20:33:30.385330+0000","last_undegraded":"2024-04-26T20:33:30.385330+0000","last_fullsized":"2024-04-26T20:33:30.385330+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:28.927822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384707+0000","last_change":"2024-04-26T20:32:26.241064+0000","last_active":"2024-04-26T20:33:30.384707+0000","last_peered":"2024-04-26T20:33:30.384707+0000","last_clean":"2024-04-26T20:33:30.384707+0000","last_became_active":"2024-04-26T20:32:26.240673+0000","last_became_peered":"2024-04-26T20:32:26.240673+0000","last_unstale":"2024-04-26T20:33:30.384707+0000","last_undegraded":"2024-04-26T20:33:30.384707+0000","last_fullsized":"2024-04-26T20:33:30.384707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:19:47.777751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383143+0000","last_change":"2024-04-26T20:32:30.388465+0000","last_active":"2024-04-26T20:33:30.383143+0000","last_peered":"2024-04-26T20:33:30.383143+0000","last_clean":"2024-04-26T20:33:30.383143+0000","last_became_active":"2024-04-26T20:32:30.388038+0000","last_became_peered":"2024-04-26T20:32:30.388038+0000","last_unstale":"2024-04-26T20:33:30.383143+0000","last_undegraded":"2024-04-26T20:33:30.383143+0000","last_fullsized":"2024-04-26T20:33:30.383143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:38:39.596970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.392793+0000","last_change":"2024-04-26T20:32:25.271914+0000","last_active":"2024-04-26T20:33:30.392793+0000","last_peered":"2024-04-26T20:33:30.392793+0000","last_clean":"2024-04-26T20:33:30.392793+0000","last_became_active":"2024-04-26T20:32:25.271567+0000","last_became_peered":"2024-04-26T20:32:25.271567+0000","last_unstale":"2024-04-26T20:33:30.392793+0000","last_undegraded":"2024-04-26T20:33:30.392793+0000","last_fullsized":"2024-04-26T20:33:30.392793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:02.182470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385349+0000","last_change":"2024-04-26T20:32:26.245344+0000","last_active":"2024-04-26T20:33:30.385349+0000","last_peered":"2024-04-26T20:33:30.385349+0000","last_clean":"2024-04-26T20:33:30.385349+0000","last_became_active":"2024-04-26T20:32:26.245032+0000","last_became_peered":"2024-04-26T20:32:26.245032+0000","last_unstale":"2024-04-26T20:33:30.385349+0000","last_undegraded":"2024-04-26T20:33:30.385349+0000","last_fullsized":"2024-04-26T20:33:30.385349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:12:51.368935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383166+0000","last_change":"2024-04-26T20:32:30.414408+0000","last_active":"2024-04-26T20:33:30.383166+0000","last_peered":"2024-04-26T20:33:30.383166+0000","last_clean":"2024-04-26T20:33:30.383166+0000","last_became_active":"2024-04-26T20:32:30.413892+0000","last_became_peered":"2024-04-26T20:32:30.413892+0000","last_unstale":"2024-04-26T20:33:30.383166+0000","last_undegraded":"2024-04-26T20:33:30.383166+0000","last_fullsized":"2024-04-26T20:33:30.383166+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:53.332840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382766+0000","last_change":"2024-04-26T20:32:25.265440+0000","last_active":"2024-04-26T20:33:30.382766+0000","last_peered":"2024-04-26T20:33:30.382766+0000","last_clean":"2024-04-26T20:33:30.382766+0000","last_became_active":"2024-04-26T20:32:25.265129+0000","last_became_peered":"2024-04-26T20:32:25.265129+0000","last_unstale":"2024-04-26T20:33:30.382766+0000","last_undegraded":"2024-04-26T20:33:30.382766+0000","last_fullsized":"2024-04-26T20:33:30.382766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:49.475009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395650+0000","last_change":"2024-04-26T20:32:26.234477+0000","last_active":"2024-04-26T20:33:30.395650+0000","last_peered":"2024-04-26T20:33:30.395650+0000","last_clean":"2024-04-26T20:33:30.395650+0000","last_became_active":"2024-04-26T20:32:26.234103+0000","last_became_peered":"2024-04-26T20:32:26.234103+0000","last_unstale":"2024-04-26T20:33:30.395650+0000","last_undegraded":"2024-04-26T20:33:30.395650+0000","last_fullsized":"2024-04-26T20:33:30.395650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:54.098860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410477+0000","last_change":"2024-04-26T20:32:30.391466+0000","last_active":"2024-04-26T20:33:30.410477+0000","last_peered":"2024-04-26T20:33:30.410477+0000","last_clean":"2024-04-26T20:33:30.410477+0000","last_became_active":"2024-04-26T20:32:30.391037+0000","last_became_peered":"2024-04-26T20:32:30.391037+0000","last_unstale":"2024-04-26T20:33:30.410477+0000","last_undegraded":"2024-04-26T20:33:30.410477+0000","last_fullsized":"2024-04-26T20:33:30.410477+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:39:16.305520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:14.827921+0000","last_change":"2024-04-26T20:32:25.281688+0000","last_active":"2024-04-26T20:34:14.827921+0000","last_peered":"2024-04-26T20:34:14.827921+0000","last_clean":"2024-04-26T20:34:14.827921+0000","last_became_active":"2024-04-26T20:32:25.281430+0000","last_became_peered":"2024-04-26T20:32:25.281430+0000","last_unstale":"2024-04-26T20:34:14.827921+0000","last_undegraded":"2024-04-26T20:34:14.827921+0000","last_fullsized":"2024-04-26T20:34:14.827921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:52.785532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395661+0000","last_change":"2024-04-26T20:32:26.226468+0000","last_active":"2024-04-26T20:33:30.395661+0000","last_peered":"2024-04-26T20:33:30.395661+0000","last_clean":"2024-04-26T20:33:30.395661+0000","last_became_active":"2024-04-26T20:32:26.225399+0000","last_became_peered":"2024-04-26T20:32:26.225399+0000","last_unstale":"2024-04-26T20:33:30.395661+0000","last_undegraded":"2024-04-26T20:33:30.395661+0000","last_fullsized":"2024-04-26T20:33:30.395661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:17:11.787658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393396+0000","last_change":"2024-04-26T20:32:30.383646+0000","last_active":"2024-04-26T20:33:30.393396+0000","last_peered":"2024-04-26T20:33:30.393396+0000","last_clean":"2024-04-26T20:33:30.393396+0000","last_became_active":"2024-04-26T20:32:30.381648+0000","last_became_peered":"2024-04-26T20:32:30.381648+0000","last_unstale":"2024-04-26T20:33:30.393396+0000","last_undegraded":"2024-04-26T20:33:30.393396+0000","last_fullsized":"2024-04-26T20:33:30.393396+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:28.036984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.392219+0000","last_change":"2024-04-26T20:32:25.275597+0000","last_active":"2024-04-26T20:33:30.392219+0000","last_peered":"2024-04-26T20:33:30.392219+0000","last_clean":"2024-04-26T20:33:30.392219+0000","last_became_active":"2024-04-26T20:32:25.275256+0000","last_became_peered":"2024-04-26T20:32:25.275256+0000","last_unstale":"2024-04-26T20:33:30.392219+0000","last_undegraded":"2024-04-26T20:33:30.392219+0000","last_fullsized":"2024-04-26T20:33:30.392219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:38:20.354879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395715+0000","last_change":"2024-04-26T20:32:26.236885+0000","last_active":"2024-04-26T20:33:30.395715+0000","last_peered":"2024-04-26T20:33:30.395715+0000","last_clean":"2024-04-26T20:33:30.395715+0000","last_became_active":"2024-04-26T20:32:26.236581+0000","last_became_peered":"2024-04-26T20:32:26.236581+0000","last_unstale":"2024-04-26T20:33:30.395715+0000","last_undegraded":"2024-04-26T20:33:30.395715+0000","last_fullsized":"2024-04-26T20:33:30.395715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:17:35.082564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391292+0000","last_change":"2024-04-26T20:32:30.380156+0000","last_active":"2024-04-26T20:33:30.391292+0000","last_peered":"2024-04-26T20:33:30.391292+0000","last_clean":"2024-04-26T20:33:30.391292+0000","last_became_active":"2024-04-26T20:32:30.379240+0000","last_became_peered":"2024-04-26T20:32:30.379240+0000","last_unstale":"2024-04-26T20:33:30.391292+0000","last_undegraded":"2024-04-26T20:33:30.391292+0000","last_fullsized":"2024-04-26T20:33:30.391292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:42:39.650923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.400793+0000","last_change":"2024-04-26T20:32:25.298850+0000","last_active":"2024-04-26T20:33:30.400793+0000","last_peered":"2024-04-26T20:33:30.400793+0000","last_clean":"2024-04-26T20:33:30.400793+0000","last_became_active":"2024-04-26T20:32:25.298171+0000","last_became_peered":"2024-04-26T20:32:25.298171+0000","last_unstale":"2024-04-26T20:33:30.400793+0000","last_undegraded":"2024-04-26T20:33:30.400793+0000","last_fullsized":"2024-04-26T20:33:30.400793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:17:03.723432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395843+0000","last_change":"2024-04-26T20:32:26.222873+0000","last_active":"2024-04-26T20:33:30.395843+0000","last_peered":"2024-04-26T20:33:30.395843+0000","last_clean":"2024-04-26T20:33:30.395843+0000","last_became_active":"2024-04-26T20:32:26.222487+0000","last_became_peered":"2024-04-26T20:32:26.222487+0000","last_unstale":"2024-04-26T20:33:30.395843+0000","last_undegraded":"2024-04-26T20:33:30.395843+0000","last_fullsized":"2024-04-26T20:33:30.395843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:17.521537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384742+0000","last_change":"2024-04-26T20:32:26.227092+0000","last_active":"2024-04-26T20:33:30.384742+0000","last_peered":"2024-04-26T20:33:30.384742+0000","last_clean":"2024-04-26T20:33:30.384742+0000","last_became_active":"2024-04-26T20:32:26.226748+0000","last_became_peered":"2024-04-26T20:32:26.226748+0000","last_unstale":"2024-04-26T20:33:30.384742+0000","last_undegraded":"2024-04-26T20:33:30.384742+0000","last_fullsized":"2024-04-26T20:33:30.384742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:25.141131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391998+0000","last_change":"2024-04-26T20:32:26.220388+0000","last_active":"2024-04-26T20:33:30.391998+0000","last_peered":"2024-04-26T20:33:30.391998+0000","last_clean":"2024-04-26T20:33:30.391998+0000","last_became_active":"2024-04-26T20:32:26.219999+0000","last_became_peered":"2024-04-26T20:32:26.219999+0000","last_unstale":"2024-04-26T20:33:30.391998+0000","last_undegraded":"2024-04-26T20:33:30.391998+0000","last_fullsized":"2024-04-26T20:33:30.391998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:11.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.392780+0000","last_change":"2024-04-26T20:32:26.240122+0000","last_active":"2024-04-26T20:33:30.392780+0000","last_peered":"2024-04-26T20:33:30.392780+0000","last_clean":"2024-04-26T20:33:30.392780+0000","last_became_active":"2024-04-26T20:32:26.239746+0000","last_became_peered":"2024-04-26T20:32:26.239746+0000","last_unstale":"2024-04-26T20:33:30.392780+0000","last_undegraded":"2024-04-26T20:33:30.392780+0000","last_fullsized":"2024-04-26T20:33:30.392780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:10.243469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394225+0000","last_change":"2024-04-26T20:32:26.254171+0000","last_active":"2024-04-26T20:33:30.394225+0000","last_peered":"2024-04-26T20:33:30.394225+0000","last_clean":"2024-04-26T20:33:30.394225+0000","last_became_active":"2024-04-26T20:32:26.253775+0000","last_became_peered":"2024-04-26T20:32:26.253775+0000","last_unstale":"2024-04-26T20:33:30.394225+0000","last_undegraded":"2024-04-26T20:33:30.394225+0000","last_fullsized":"2024-04-26T20:33:30.394225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:19:57.968344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.399014+0000","last_change":"2024-04-26T20:32:26.256475+0000","last_active":"2024-04-26T20:33:30.399014+0000","last_peered":"2024-04-26T20:33:30.399014+0000","last_clean":"2024-04-26T20:33:30.399014+0000","last_became_active":"2024-04-26T20:32:26.256119+0000","last_became_peered":"2024-04-26T20:32:26.256119+0000","last_unstale":"2024-04-26T20:33:30.399014+0000","last_undegraded":"2024-04-26T20:33:30.399014+0000","last_fullsized":"2024-04-26T20:33:30.399014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:39:31.456077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385136+0000","last_change":"2024-04-26T20:32:26.229251+0000","last_active":"2024-04-26T20:33:30.385136+0000","last_peered":"2024-04-26T20:33:30.385136+0000","last_clean":"2024-04-26T20:33:30.385136+0000","last_became_active":"2024-04-26T20:32:26.228847+0000","last_became_peered":"2024-04-26T20:32:26.228847+0000","last_unstale":"2024-04-26T20:33:30.385136+0000","last_undegraded":"2024-04-26T20:33:30.385136+0000","last_fullsized":"2024-04-26T20:33:30.385136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:07:28.489681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.392696+0000","last_change":"2024-04-26T20:32:26.223939+0000","last_active":"2024-04-26T20:33:30.392696+0000","last_peered":"2024-04-26T20:33:30.392696+0000","last_clean":"2024-04-26T20:33:30.392696+0000","last_became_active":"2024-04-26T20:32:26.223602+0000","last_became_peered":"2024-04-26T20:32:26.223602+0000","last_unstale":"2024-04-26T20:33:30.392696+0000","last_undegraded":"2024-04-26T20:33:30.392696+0000","last_fullsized":"2024-04-26T20:33:30.392696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:54.588411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.389339+0000","last_change":"2024-04-26T20:32:26.229912+0000","last_active":"2024-04-26T20:33:30.389339+0000","last_peered":"2024-04-26T20:33:30.389339+0000","last_clean":"2024-04-26T20:33:30.389339+0000","last_became_active":"2024-04-26T20:32:26.228498+0000","last_became_peered":"2024-04-26T20:32:26.228498+0000","last_unstale":"2024-04-26T20:33:30.389339+0000","last_undegraded":"2024-04-26T20:33:30.389339+0000","last_fullsized":"2024-04-26T20:33:30.389339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:17:53.302225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391674+0000","last_change":"2024-04-26T20:32:26.232087+0000","last_active":"2024-04-26T20:33:30.391674+0000","last_peered":"2024-04-26T20:33:30.391674+0000","last_clean":"2024-04-26T20:33:30.391674+0000","last_became_active":"2024-04-26T20:32:26.231525+0000","last_became_peered":"2024-04-26T20:32:26.231525+0000","last_unstale":"2024-04-26T20:33:30.391674+0000","last_undegraded":"2024-04-26T20:33:30.391674+0000","last_fullsized":"2024-04-26T20:33:30.391674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:05.326866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385625+0000","last_change":"2024-04-26T20:32:26.224341+0000","last_active":"2024-04-26T20:33:30.385625+0000","last_peered":"2024-04-26T20:33:30.385625+0000","last_clean":"2024-04-26T20:33:30.385625+0000","last_became_active":"2024-04-26T20:32:26.223888+0000","last_became_peered":"2024-04-26T20:32:26.223888+0000","last_unstale":"2024-04-26T20:33:30.385625+0000","last_undegraded":"2024-04-26T20:33:30.385625+0000","last_fullsized":"2024-04-26T20:33:30.385625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:21.103180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410671+0000","last_change":"2024-04-26T20:32:26.231255+0000","last_active":"2024-04-26T20:33:30.410671+0000","last_peered":"2024-04-26T20:33:30.410671+0000","last_clean":"2024-04-26T20:33:30.410671+0000","last_became_active":"2024-04-26T20:32:26.230823+0000","last_became_peered":"2024-04-26T20:32:26.230823+0000","last_unstale":"2024-04-26T20:33:30.410671+0000","last_undegraded":"2024-04-26T20:33:30.410671+0000","last_fullsized":"2024-04-26T20:33:30.410671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:34.066151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.397785+0000","last_change":"2024-04-26T20:32:26.246452+0000","last_active":"2024-04-26T20:33:30.397785+0000","last_peered":"2024-04-26T20:33:30.397785+0000","last_clean":"2024-04-26T20:33:30.397785+0000","last_became_active":"2024-04-26T20:32:26.246109+0000","last_became_peered":"2024-04-26T20:32:26.246109+0000","last_unstale":"2024-04-26T20:33:30.397785+0000","last_undegraded":"2024-04-26T20:33:30.397785+0000","last_fullsized":"2024-04-26T20:33:30.397785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:42:21.617776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.405491+0000","last_change":"2024-04-26T20:33:30.405491+0000","last_active":"2024-04-26T20:33:30.405491+0000","last_peered":"2024-04-26T20:33:30.405491+0000","last_clean":"2024-04-26T20:33:30.405491+0000","last_became_active":"2024-04-26T20:33:30.405010+0000","last_became_peered":"2024-04-26T20:33:30.405010+0000","last_unstale":"2024-04-26T20:33:30.405491+0000","last_undegraded":"2024-04-26T20:33:30.405491+0000","last_fullsized":"2024-04-26T20:33:30.405491+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:23:13.675737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61248,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":847,"num_read_kb":494,"num_write":244,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":501449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28704,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716448,"statfs":{"total":95995035648,"available":95965642752,"internally_reserved":0,"allocated":1720320,"data_stored":1143091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808879,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":505475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":501757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1650688,"data_stored":1097863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":496883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":552739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":498339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":552337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":496591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":501579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1658880,"data_stored":1093445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":56578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":56176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":56562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T20:34:20.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T20:34:21.313 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:21.748 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:21 smithi102 ceph-mon[28094]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:34:21.748 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:21 smithi102 ceph-mon[28094]: pgmap v377: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:34:22.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:21 smithi112 ceph-mon[32769]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:34:22.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:21 smithi112 ceph-mon[32769]: pgmap v377: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:34:22.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:21 smithi130 ceph-mon[32704]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:34:22.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:21 smithi130 ceph-mon[32704]: pgmap v377: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:34:23.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.431+0000 7f08ad400700 1 -- 172.21.15.102:0/2408412848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08a8078270 msgr2=0x7f08a8113c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:23.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.431+0000 7f08ad400700 1 --2- 172.21.15.102:0/2408412848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08a8078270 0x7f08a8113c90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0894009990 tx=0x7f0894009ca0 comp rx=0 tx=0).stop 2024-04-26T20:34:23.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.431+0000 7f08ad400700 1 -- 172.21.15.102:0/2408412848 shutdown_connections 2024-04-26T20:34:23.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.431+0000 7f08ad400700 1 --2- 172.21.15.102:0/2408412848 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08a81141d0 0x7f08a81165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:23.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.431+0000 7f08ad400700 1 --2- 172.21.15.102:0/2408412848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08a8077950 0x7f08a8077d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:23.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.431+0000 7f08ad400700 1 --2- 172.21.15.102:0/2408412848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08a8078270 0x7f08a8113c90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:23.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.431+0000 7f08ad400700 1 -- 172.21.15.102:0/2408412848 >> 172.21.15.102:0/2408412848 conn(0x7f08a8101060 msgr2=0x7f08a8103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:23.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.432+0000 7f08ad400700 1 -- 172.21.15.102:0/2408412848 shutdown_connections 2024-04-26T20:34:23.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.432+0000 7f08ad400700 1 -- 172.21.15.102:0/2408412848 wait complete. 2024-04-26T20:34:23.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08ad400700 1 Processor -- start 2024-04-26T20:34:23.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08ad400700 1 -- start start 2024-04-26T20:34:23.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08ad400700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08a8077950 0x7f08a81a00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:23.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08ad400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08a8078270 0x7f08a81a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:23.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08ad400700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08a81141d0 0x7f08a81a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:23.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08ad400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f08a8119b50 con 0x7f08a8078270 2024-04-26T20:34:23.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08ad400700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f08a81199d0 con 0x7f08a8077950 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08ad400700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f08a8119cd0 con 0x7f08a81141d0 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08a7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08a8078270 0x7f08a81a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08a7e00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08a8077950 0x7f08a81a00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.433+0000 7f08a7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08a8078270 0x7f08a81a0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34626/0 (socket says 172.21.15.102:34626) 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.434+0000 7f08a7400700 1 -- 172.21.15.102:0/2401301835 learned_addr learned my addr 172.21.15.102:0/2401301835 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.434+0000 7f08aca00700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08a81141d0 0x7f08a81a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.434+0000 7f08a7e00700 1 -- 172.21.15.102:0/2401301835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08a81141d0 msgr2=0x7f08a81a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.434+0000 7f08a7e00700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08a81141d0 0x7f08a81a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.434+0000 7f08a7e00700 1 -- 172.21.15.102:0/2401301835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08a8078270 msgr2=0x7f08a81a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.434+0000 7f08a7e00700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08a8078270 0x7f08a81a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.434+0000 7f08a7e00700 1 -- 172.21.15.102:0/2401301835 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a81a50d0 con 0x7f08a8077950 2024-04-26T20:34:23.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.434+0000 7f08a7e00700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08a8077950 0x7f08a81a00c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f089800e9f0 tx=0x7f089800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:23.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.434+0000 7f08a4c00700 1 -- 172.21.15.102:0/2401301835 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089800cb70 con 0x7f08a8077950 2024-04-26T20:34:23.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.435+0000 7f08ad400700 1 -- 172.21.15.102:0/2401301835 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a81a53c0 con 0x7f08a8077950 2024-04-26T20:34:23.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.435+0000 7f08a4c00700 1 -- 172.21.15.102:0/2401301835 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f089800ee90 con 0x7f08a8077950 2024-04-26T20:34:23.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.435+0000 7f08a4c00700 1 -- 172.21.15.102:0/2401301835 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898003d40 con 0x7f08a8077950 2024-04-26T20:34:23.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.435+0000 7f08ad400700 1 -- 172.21.15.102:0/2401301835 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a81acc40 con 0x7f08a8077950 2024-04-26T20:34:23.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.436+0000 7f08ad400700 1 -- 172.21.15.102:0/2401301835 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a806a110 con 0x7f08a8077950 2024-04-26T20:34:23.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.438+0000 7f08a4c00700 1 -- 172.21.15.102:0/2401301835 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f0898003ee0 con 0x7f08a8077950 2024-04-26T20:34:23.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.439+0000 7f08a4c00700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f088c0a3d50 0x7f088c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:23.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.439+0000 7f08a4c00700 1 -- 172.21.15.102:0/2401301835 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f08980d09a0 con 0x7f08a8077950 2024-04-26T20:34:23.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.440+0000 7f08a7400700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f088c0a3d50 0x7f088c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:23.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.440+0000 7f08a7400700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f088c0a3d50 0x7f088c0a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0894005cd0 tx=0x7f0894005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:23.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.443+0000 7f08a4c00700 1 -- 172.21.15.102:0/2401301835 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f089809ce60 con 0x7f08a8077950 2024-04-26T20:34:23.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.665+0000 7f08ad400700 1 -- 172.21.15.102:0/2401301835 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f08a81a1420 con 0x7f088c0a3d50 2024-04-26T20:34:23.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.702+0000 7f08a4c00700 1 -- 172.21.15.102:0/2401301835 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340388 (secure 0 0 0) 0x7f08a81a1420 con 0x7f088c0a3d50 2024-04-26T20:34:23.710 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.710+0000 7f08ad400700 1 -- 172.21.15.102:0/2401301835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f088c0a3d50 msgr2=0x7f088c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f088c0a3d50 0x7f088c0a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0894005cd0 tx=0x7f0894005c60 comp rx=0 tx=0).stop 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 -- 172.21.15.102:0/2401301835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08a8077950 msgr2=0x7f08a81a00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08a8077950 0x7f08a81a00c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f089800e9f0 tx=0x7f089800ed00 comp rx=0 tx=0).stop 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 -- 172.21.15.102:0/2401301835 shutdown_connections 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f088c0a3d50 0x7f088c0a6210 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08a81141d0 0x7f08a81a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f08a8077950 0x7f08a81a00c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 --2- 172.21.15.102:0/2401301835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08a8078270 0x7f08a81a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 -- 172.21.15.102:0/2401301835 >> 172.21.15.102:0/2401301835 conn(0x7f08a8101060 msgr2=0x7f08a8103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:23.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 -- 172.21.15.102:0/2401301835 shutdown_connections 2024-04-26T20:34:23.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:23.711+0000 7f08ad400700 1 -- 172.21.15.102:0/2401301835 wait complete. 2024-04-26T20:34:23.714 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-26T20:34:24.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:23 smithi112 ceph-mon[32769]: pgmap v378: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:34:24.093 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:23 smithi102 ceph-mon[28094]: pgmap v378: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:34:24.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:23 smithi130 ceph-mon[32704]: pgmap v378: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:34:24.592 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-04-26T20:34:22.565209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651850,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":993,"num_read_kb":608,"num_write":427,"num_write_kb":3073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181358,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":350964,"kb_used_data":14216,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124590860,"statfs":{"total":1151940427776,"available":1151581040640,"internally_reserved":0,"allocated":14557184,"data_stored":7941548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":344634682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":49,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005767"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385476+0000","last_change":"2024-04-26T20:32:26.249504+0000","last_active":"2024-04-26T20:33:30.385476+0000","last_peered":"2024-04-26T20:33:30.385476+0000","last_clean":"2024-04-26T20:33:30.385476+0000","last_became_active":"2024-04-26T20:32:26.249188+0000","last_became_peered":"2024-04-26T20:32:26.249188+0000","last_unstale":"2024-04-26T20:33:30.385476+0000","last_undegraded":"2024-04-26T20:33:30.385476+0000","last_fullsized":"2024-04-26T20:33:30.385476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:28:15.906199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394564+0000","last_change":"2024-04-26T20:32:26.246991+0000","last_active":"2024-04-26T20:33:30.394564+0000","last_peered":"2024-04-26T20:33:30.394564+0000","last_clean":"2024-04-26T20:33:30.394564+0000","last_became_active":"2024-04-26T20:32:26.245552+0000","last_became_peered":"2024-04-26T20:32:26.245552+0000","last_unstale":"2024-04-26T20:33:30.394564+0000","last_undegraded":"2024-04-26T20:33:30.394564+0000","last_fullsized":"2024-04-26T20:33:30.394564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:59:46.487659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394679+0000","last_change":"2024-04-26T20:32:26.241451+0000","last_active":"2024-04-26T20:33:30.394679+0000","last_peered":"2024-04-26T20:33:30.394679+0000","last_clean":"2024-04-26T20:33:30.394679+0000","last_became_active":"2024-04-26T20:32:26.240012+0000","last_became_peered":"2024-04-26T20:32:26.240012+0000","last_unstale":"2024-04-26T20:33:30.394679+0000","last_undegraded":"2024-04-26T20:33:30.394679+0000","last_fullsized":"2024-04-26T20:33:30.394679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:43:12.776082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384131+0000","last_change":"2024-04-26T20:32:26.224138+0000","last_active":"2024-04-26T20:33:30.384131+0000","last_peered":"2024-04-26T20:33:30.384131+0000","last_clean":"2024-04-26T20:33:30.384131+0000","last_became_active":"2024-04-26T20:32:26.223827+0000","last_became_peered":"2024-04-26T20:32:26.223827+0000","last_unstale":"2024-04-26T20:33:30.384131+0000","last_undegraded":"2024-04-26T20:33:30.384131+0000","last_fullsized":"2024-04-26T20:33:30.384131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:38:33.323001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382762+0000","last_change":"2024-04-26T20:32:26.240298+0000","last_active":"2024-04-26T20:33:30.382762+0000","last_peered":"2024-04-26T20:33:30.382762+0000","last_clean":"2024-04-26T20:33:30.382762+0000","last_became_active":"2024-04-26T20:32:26.239933+0000","last_became_peered":"2024-04-26T20:32:26.239933+0000","last_unstale":"2024-04-26T20:33:30.382762+0000","last_undegraded":"2024-04-26T20:33:30.382762+0000","last_fullsized":"2024-04-26T20:33:30.382762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:37:00.618136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.406074+0000","last_change":"2024-04-26T20:33:30.406074+0000","last_active":"2024-04-26T20:33:30.406074+0000","last_peered":"2024-04-26T20:33:30.406074+0000","last_clean":"2024-04-26T20:33:30.406074+0000","last_became_active":"2024-04-26T20:33:30.405818+0000","last_became_peered":"2024-04-26T20:33:30.405818+0000","last_unstale":"2024-04-26T20:33:30.406074+0000","last_undegraded":"2024-04-26T20:33:30.406074+0000","last_fullsized":"2024-04-26T20:33:30.406074+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:15:38.754348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395078+0000","last_change":"2024-04-26T20:32:26.232600+0000","last_active":"2024-04-26T20:33:30.395078+0000","last_peered":"2024-04-26T20:33:30.395078+0000","last_clean":"2024-04-26T20:33:30.395078+0000","last_became_active":"2024-04-26T20:32:26.232320+0000","last_became_peered":"2024-04-26T20:32:26.232320+0000","last_unstale":"2024-04-26T20:33:30.395078+0000","last_undegraded":"2024-04-26T20:33:30.395078+0000","last_fullsized":"2024-04-26T20:33:30.395078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:18.199940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.405729+0000","last_change":"2024-04-26T20:33:30.405729+0000","last_active":"2024-04-26T20:33:30.405729+0000","last_peered":"2024-04-26T20:33:30.405729+0000","last_clean":"2024-04-26T20:33:30.405729+0000","last_became_active":"2024-04-26T20:33:30.404668+0000","last_became_peered":"2024-04-26T20:33:30.404668+0000","last_unstale":"2024-04-26T20:33:30.405729+0000","last_undegraded":"2024-04-26T20:33:30.405729+0000","last_fullsized":"2024-04-26T20:33:30.405729+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:21:59.224049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.411139+0000","last_change":"2024-04-26T20:32:26.230224+0000","last_active":"2024-04-26T20:33:30.411139+0000","last_peered":"2024-04-26T20:33:30.411139+0000","last_clean":"2024-04-26T20:33:30.411139+0000","last_became_active":"2024-04-26T20:32:26.229865+0000","last_became_peered":"2024-04-26T20:32:26.229865+0000","last_unstale":"2024-04-26T20:33:30.411139+0000","last_undegraded":"2024-04-26T20:33:30.411139+0000","last_fullsized":"2024-04-26T20:33:30.411139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:03.060925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.386331+0000","last_change":"2024-04-26T20:32:26.219841+0000","last_active":"2024-04-26T20:33:30.386331+0000","last_peered":"2024-04-26T20:33:30.386331+0000","last_clean":"2024-04-26T20:33:30.386331+0000","last_became_active":"2024-04-26T20:32:26.219505+0000","last_became_peered":"2024-04-26T20:32:26.219505+0000","last_unstale":"2024-04-26T20:33:30.386331+0000","last_undegraded":"2024-04-26T20:33:30.386331+0000","last_fullsized":"2024-04-26T20:33:30.386331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:11.514510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395570+0000","last_change":"2024-04-26T20:32:26.226427+0000","last_active":"2024-04-26T20:33:30.395570+0000","last_peered":"2024-04-26T20:33:30.395570+0000","last_clean":"2024-04-26T20:33:30.395570+0000","last_became_active":"2024-04-26T20:32:26.226112+0000","last_became_peered":"2024-04-26T20:32:26.226112+0000","last_unstale":"2024-04-26T20:33:30.395570+0000","last_undegraded":"2024-04-26T20:33:30.395570+0000","last_fullsized":"2024-04-26T20:33:30.395570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:56.506369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384019+0000","last_change":"2024-04-26T20:32:26.222756+0000","last_active":"2024-04-26T20:33:30.384019+0000","last_peered":"2024-04-26T20:33:30.384019+0000","last_clean":"2024-04-26T20:33:30.384019+0000","last_became_active":"2024-04-26T20:32:26.222342+0000","last_became_peered":"2024-04-26T20:32:26.222342+0000","last_unstale":"2024-04-26T20:33:30.384019+0000","last_undegraded":"2024-04-26T20:33:30.384019+0000","last_fullsized":"2024-04-26T20:33:30.384019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:45:11.543434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.407864+0000","last_change":"2024-04-26T20:33:30.407864+0000","last_active":"2024-04-26T20:33:30.407864+0000","last_peered":"2024-04-26T20:33:30.407864+0000","last_clean":"2024-04-26T20:33:30.407864+0000","last_became_active":"2024-04-26T20:33:30.407252+0000","last_became_peered":"2024-04-26T20:33:30.407252+0000","last_unstale":"2024-04-26T20:33:30.407864+0000","last_undegraded":"2024-04-26T20:33:30.407864+0000","last_fullsized":"2024-04-26T20:33:30.407864+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:28.664781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393728+0000","last_change":"2024-04-26T20:32:26.247783+0000","last_active":"2024-04-26T20:33:30.393728+0000","last_peered":"2024-04-26T20:33:30.393728+0000","last_clean":"2024-04-26T20:33:30.393728+0000","last_became_active":"2024-04-26T20:32:26.247466+0000","last_became_peered":"2024-04-26T20:32:26.247466+0000","last_unstale":"2024-04-26T20:33:30.393728+0000","last_undegraded":"2024-04-26T20:33:30.393728+0000","last_fullsized":"2024-04-26T20:33:30.393728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:36:59.424377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393596+0000","last_change":"2024-04-26T20:32:26.249823+0000","last_active":"2024-04-26T20:33:30.393596+0000","last_peered":"2024-04-26T20:33:30.393596+0000","last_clean":"2024-04-26T20:33:30.393596+0000","last_became_active":"2024-04-26T20:32:26.248954+0000","last_became_peered":"2024-04-26T20:32:26.248954+0000","last_unstale":"2024-04-26T20:33:30.393596+0000","last_undegraded":"2024-04-26T20:33:30.393596+0000","last_fullsized":"2024-04-26T20:33:30.393596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:55:44.246537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410577+0000","last_change":"2024-04-26T20:32:26.241997+0000","last_active":"2024-04-26T20:33:30.410577+0000","last_peered":"2024-04-26T20:33:30.410577+0000","last_clean":"2024-04-26T20:33:30.410577+0000","last_became_active":"2024-04-26T20:32:26.240856+0000","last_became_peered":"2024-04-26T20:32:26.240856+0000","last_unstale":"2024-04-26T20:33:30.410577+0000","last_undegraded":"2024-04-26T20:33:30.410577+0000","last_fullsized":"2024-04-26T20:33:30.410577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:52:53.895994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385301+0000","last_change":"2024-04-26T20:32:26.241097+0000","last_active":"2024-04-26T20:33:30.385301+0000","last_peered":"2024-04-26T20:33:30.385301+0000","last_clean":"2024-04-26T20:33:30.385301+0000","last_became_active":"2024-04-26T20:32:26.240770+0000","last_became_peered":"2024-04-26T20:32:26.240770+0000","last_unstale":"2024-04-26T20:33:30.385301+0000","last_undegraded":"2024-04-26T20:33:30.385301+0000","last_fullsized":"2024-04-26T20:33:30.385301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:10.443314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.402985+0000","last_change":"2024-04-26T20:32:26.232279+0000","last_active":"2024-04-26T20:33:30.402985+0000","last_peered":"2024-04-26T20:33:30.402985+0000","last_clean":"2024-04-26T20:33:30.402985+0000","last_became_active":"2024-04-26T20:32:26.231256+0000","last_became_peered":"2024-04-26T20:32:26.231256+0000","last_unstale":"2024-04-26T20:33:30.402985+0000","last_undegraded":"2024-04-26T20:33:30.402985+0000","last_fullsized":"2024-04-26T20:33:30.402985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:26.484932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393047+0000","last_change":"2024-04-26T20:32:26.249876+0000","last_active":"2024-04-26T20:33:30.393047+0000","last_peered":"2024-04-26T20:33:30.393047+0000","last_clean":"2024-04-26T20:33:30.393047+0000","last_became_active":"2024-04-26T20:32:26.249416+0000","last_became_peered":"2024-04-26T20:32:26.249416+0000","last_unstale":"2024-04-26T20:33:30.393047+0000","last_undegraded":"2024-04-26T20:33:30.393047+0000","last_fullsized":"2024-04-26T20:33:30.393047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:03:54.616941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391854+0000","last_change":"2024-04-26T20:32:30.389128+0000","last_active":"2024-04-26T20:33:30.391854+0000","last_peered":"2024-04-26T20:33:30.391854+0000","last_clean":"2024-04-26T20:33:30.391854+0000","last_became_active":"2024-04-26T20:32:30.388668+0000","last_became_peered":"2024-04-26T20:32:30.388668+0000","last_unstale":"2024-04-26T20:33:30.391854+0000","last_undegraded":"2024-04-26T20:33:30.391854+0000","last_fullsized":"2024-04-26T20:33:30.391854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:12.019691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":126,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:18.889174+0000","last_change":"2024-04-26T20:32:25.278417+0000","last_active":"2024-04-26T20:34:18.889174+0000","last_peered":"2024-04-26T20:34:18.889174+0000","last_clean":"2024-04-26T20:34:18.889174+0000","last_became_active":"2024-04-26T20:32:25.278004+0000","last_became_peered":"2024-04-26T20:32:25.278004+0000","last_unstale":"2024-04-26T20:34:18.889174+0000","last_undegraded":"2024-04-26T20:34:18.889174+0000","last_fullsized":"2024-04-26T20:34:18.889174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:51.458285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.387891+0000","last_change":"2024-04-26T20:32:26.229015+0000","last_active":"2024-04-26T20:33:30.387891+0000","last_peered":"2024-04-26T20:33:30.387891+0000","last_clean":"2024-04-26T20:33:30.387891+0000","last_became_active":"2024-04-26T20:32:26.228729+0000","last_became_peered":"2024-04-26T20:32:26.228729+0000","last_unstale":"2024-04-26T20:33:30.387891+0000","last_undegraded":"2024-04-26T20:33:30.387891+0000","last_fullsized":"2024-04-26T20:33:30.387891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:20.014133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382866+0000","last_change":"2024-04-26T20:32:30.383196+0000","last_active":"2024-04-26T20:33:30.382866+0000","last_peered":"2024-04-26T20:33:30.382866+0000","last_clean":"2024-04-26T20:33:30.382866+0000","last_became_active":"2024-04-26T20:32:30.382831+0000","last_became_peered":"2024-04-26T20:32:30.382831+0000","last_unstale":"2024-04-26T20:33:30.382866+0000","last_undegraded":"2024-04-26T20:33:30.382866+0000","last_fullsized":"2024-04-26T20:33:30.382866+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:48:13.696394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.390035+0000","last_change":"2024-04-26T20:32:25.294227+0000","last_active":"2024-04-26T20:33:30.390035+0000","last_peered":"2024-04-26T20:33:30.390035+0000","last_clean":"2024-04-26T20:33:30.390035+0000","last_became_active":"2024-04-26T20:32:25.287681+0000","last_became_peered":"2024-04-26T20:32:25.287681+0000","last_unstale":"2024-04-26T20:33:30.390035+0000","last_undegraded":"2024-04-26T20:33:30.390035+0000","last_fullsized":"2024-04-26T20:33:30.390035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:00:25.567135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395225+0000","last_change":"2024-04-26T20:32:26.251364+0000","last_active":"2024-04-26T20:33:30.395225+0000","last_peered":"2024-04-26T20:33:30.395225+0000","last_clean":"2024-04-26T20:33:30.395225+0000","last_became_active":"2024-04-26T20:32:26.251037+0000","last_became_peered":"2024-04-26T20:32:26.251037+0000","last_unstale":"2024-04-26T20:33:30.395225+0000","last_undegraded":"2024-04-26T20:33:30.395225+0000","last_fullsized":"2024-04-26T20:33:30.395225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:17:41.813889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394688+0000","last_change":"2024-04-26T20:33:30.394688+0000","last_active":"2024-04-26T20:33:30.394688+0000","last_peered":"2024-04-26T20:33:30.394688+0000","last_clean":"2024-04-26T20:33:30.394688+0000","last_became_active":"2024-04-26T20:33:30.394199+0000","last_became_peered":"2024-04-26T20:33:30.394199+0000","last_unstale":"2024-04-26T20:33:30.394688+0000","last_undegraded":"2024-04-26T20:33:30.394688+0000","last_fullsized":"2024-04-26T20:33:30.394688+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:09:25.418017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383314+0000","last_change":"2024-04-26T20:32:25.267156+0000","last_active":"2024-04-26T20:33:30.383314+0000","last_peered":"2024-04-26T20:33:30.383314+0000","last_clean":"2024-04-26T20:33:30.383314+0000","last_became_active":"2024-04-26T20:32:25.266113+0000","last_became_peered":"2024-04-26T20:32:25.266113+0000","last_unstale":"2024-04-26T20:33:30.383314+0000","last_undegraded":"2024-04-26T20:33:30.383314+0000","last_fullsized":"2024-04-26T20:33:30.383314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:30:38.798437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410697+0000","last_change":"2024-04-26T20:32:26.249908+0000","last_active":"2024-04-26T20:33:30.410697+0000","last_peered":"2024-04-26T20:33:30.410697+0000","last_clean":"2024-04-26T20:33:30.410697+0000","last_became_active":"2024-04-26T20:32:26.249576+0000","last_became_peered":"2024-04-26T20:32:26.249576+0000","last_unstale":"2024-04-26T20:33:30.410697+0000","last_undegraded":"2024-04-26T20:33:30.410697+0000","last_fullsized":"2024-04-26T20:33:30.410697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:37:09.611624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382867+0000","last_change":"2024-04-26T20:32:30.391456+0000","last_active":"2024-04-26T20:33:30.382867+0000","last_peered":"2024-04-26T20:33:30.382867+0000","last_clean":"2024-04-26T20:33:30.382867+0000","last_became_active":"2024-04-26T20:32:30.391028+0000","last_became_peered":"2024-04-26T20:32:30.391028+0000","last_unstale":"2024-04-26T20:33:30.382867+0000","last_undegraded":"2024-04-26T20:33:30.382867+0000","last_fullsized":"2024-04-26T20:33:30.382867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:55:17.987421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383050+0000","last_change":"2024-04-26T20:32:25.264519+0000","last_active":"2024-04-26T20:33:30.383050+0000","last_peered":"2024-04-26T20:33:30.383050+0000","last_clean":"2024-04-26T20:33:30.383050+0000","last_became_active":"2024-04-26T20:32:25.264189+0000","last_became_peered":"2024-04-26T20:32:25.264189+0000","last_unstale":"2024-04-26T20:33:30.383050+0000","last_undegraded":"2024-04-26T20:33:30.383050+0000","last_fullsized":"2024-04-26T20:33:30.383050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:12:56.167119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391829+0000","last_change":"2024-04-26T20:32:26.226281+0000","last_active":"2024-04-26T20:33:30.391829+0000","last_peered":"2024-04-26T20:33:30.391829+0000","last_clean":"2024-04-26T20:33:30.391829+0000","last_became_active":"2024-04-26T20:32:26.225958+0000","last_became_peered":"2024-04-26T20:32:26.225958+0000","last_unstale":"2024-04-26T20:33:30.391829+0000","last_undegraded":"2024-04-26T20:33:30.391829+0000","last_fullsized":"2024-04-26T20:33:30.391829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:22:30.914293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391769+0000","last_change":"2024-04-26T20:32:30.379734+0000","last_active":"2024-04-26T20:33:30.391769+0000","last_peered":"2024-04-26T20:33:30.391769+0000","last_clean":"2024-04-26T20:33:30.391769+0000","last_became_active":"2024-04-26T20:32:30.379240+0000","last_became_peered":"2024-04-26T20:32:30.379240+0000","last_unstale":"2024-04-26T20:33:30.391769+0000","last_undegraded":"2024-04-26T20:33:30.391769+0000","last_fullsized":"2024-04-26T20:33:30.391769+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:05:46.669583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":129,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:17.461403+0000","last_change":"2024-04-26T20:33:30.400010+0000","last_active":"2024-04-26T20:34:17.461403+0000","last_peered":"2024-04-26T20:34:17.461403+0000","last_clean":"2024-04-26T20:34:17.461403+0000","last_became_active":"2024-04-26T20:33:30.396163+0000","last_became_peered":"2024-04-26T20:33:30.396163+0000","last_unstale":"2024-04-26T20:34:17.461403+0000","last_undegraded":"2024-04-26T20:34:17.461403+0000","last_fullsized":"2024-04-26T20:34:17.461403+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:43:47.524772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382753+0000","last_change":"2024-04-26T20:32:26.235743+0000","last_active":"2024-04-26T20:33:30.382753+0000","last_peered":"2024-04-26T20:33:30.382753+0000","last_clean":"2024-04-26T20:33:30.382753+0000","last_became_active":"2024-04-26T20:32:26.235375+0000","last_became_peered":"2024-04-26T20:32:26.235375+0000","last_unstale":"2024-04-26T20:33:30.382753+0000","last_undegraded":"2024-04-26T20:33:30.382753+0000","last_fullsized":"2024-04-26T20:33:30.382753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:27.673837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.403841+0000","last_change":"2024-04-26T20:33:30.403841+0000","last_active":"2024-04-26T20:33:30.403841+0000","last_peered":"2024-04-26T20:33:30.403841+0000","last_clean":"2024-04-26T20:33:30.403841+0000","last_became_active":"2024-04-26T20:33:30.403355+0000","last_became_peered":"2024-04-26T20:33:30.403355+0000","last_unstale":"2024-04-26T20:33:30.403841+0000","last_undegraded":"2024-04-26T20:33:30.403841+0000","last_fullsized":"2024-04-26T20:33:30.403841+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:27.784185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385778+0000","last_change":"2024-04-26T20:32:25.301063+0000","last_active":"2024-04-26T20:33:30.385778+0000","last_peered":"2024-04-26T20:33:30.385778+0000","last_clean":"2024-04-26T20:33:30.385778+0000","last_became_active":"2024-04-26T20:32:25.300755+0000","last_became_peered":"2024-04-26T20:32:25.300755+0000","last_unstale":"2024-04-26T20:33:30.385778+0000","last_undegraded":"2024-04-26T20:33:30.385778+0000","last_fullsized":"2024-04-26T20:33:30.385778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:50.542961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393168+0000","last_change":"2024-04-26T20:32:26.223507+0000","last_active":"2024-04-26T20:33:30.393168+0000","last_peered":"2024-04-26T20:33:30.393168+0000","last_clean":"2024-04-26T20:33:30.393168+0000","last_became_active":"2024-04-26T20:32:26.222991+0000","last_became_peered":"2024-04-26T20:32:26.222991+0000","last_unstale":"2024-04-26T20:33:30.393168+0000","last_undegraded":"2024-04-26T20:33:30.393168+0000","last_fullsized":"2024-04-26T20:33:30.393168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:53:25.207884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385297+0000","last_change":"2024-04-26T20:32:30.389354+0000","last_active":"2024-04-26T20:33:30.385297+0000","last_peered":"2024-04-26T20:33:30.385297+0000","last_clean":"2024-04-26T20:33:30.385297+0000","last_became_active":"2024-04-26T20:32:30.388848+0000","last_became_peered":"2024-04-26T20:32:30.388848+0000","last_unstale":"2024-04-26T20:33:30.385297+0000","last_undegraded":"2024-04-26T20:33:30.385297+0000","last_fullsized":"2024-04-26T20:33:30.385297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:12.615750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.399639+0000","last_change":"2024-04-26T20:32:25.292694+0000","last_active":"2024-04-26T20:33:30.399639+0000","last_peered":"2024-04-26T20:33:30.399639+0000","last_clean":"2024-04-26T20:33:30.399639+0000","last_became_active":"2024-04-26T20:32:25.283757+0000","last_became_peered":"2024-04-26T20:32:25.283757+0000","last_unstale":"2024-04-26T20:33:30.399639+0000","last_undegraded":"2024-04-26T20:33:30.399639+0000","last_fullsized":"2024-04-26T20:33:30.399639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:05:15.294842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384573+0000","last_change":"2024-04-26T20:32:26.232770+0000","last_active":"2024-04-26T20:33:30.384573+0000","last_peered":"2024-04-26T20:33:30.384573+0000","last_clean":"2024-04-26T20:33:30.384573+0000","last_became_active":"2024-04-26T20:32:26.232381+0000","last_became_peered":"2024-04-26T20:32:26.232381+0000","last_unstale":"2024-04-26T20:33:30.384573+0000","last_undegraded":"2024-04-26T20:33:30.384573+0000","last_fullsized":"2024-04-26T20:33:30.384573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:16:43.310582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394108+0000","last_change":"2024-04-26T20:32:30.389690+0000","last_active":"2024-04-26T20:33:30.394108+0000","last_peered":"2024-04-26T20:33:30.394108+0000","last_clean":"2024-04-26T20:33:30.394108+0000","last_became_active":"2024-04-26T20:32:30.389258+0000","last_became_peered":"2024-04-26T20:32:30.389258+0000","last_unstale":"2024-04-26T20:33:30.394108+0000","last_undegraded":"2024-04-26T20:33:30.394108+0000","last_fullsized":"2024-04-26T20:33:30.394108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:36.528757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391527+0000","last_change":"2024-04-26T20:32:25.267681+0000","last_active":"2024-04-26T20:33:30.391527+0000","last_peered":"2024-04-26T20:33:30.391527+0000","last_clean":"2024-04-26T20:33:30.391527+0000","last_became_active":"2024-04-26T20:32:25.267351+0000","last_became_peered":"2024-04-26T20:32:25.267351+0000","last_unstale":"2024-04-26T20:33:30.391527+0000","last_undegraded":"2024-04-26T20:33:30.391527+0000","last_fullsized":"2024-04-26T20:33:30.391527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:00:26.339458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393346+0000","last_change":"2024-04-26T20:32:26.243638+0000","last_active":"2024-04-26T20:33:30.393346+0000","last_peered":"2024-04-26T20:33:30.393346+0000","last_clean":"2024-04-26T20:33:30.393346+0000","last_became_active":"2024-04-26T20:32:26.241783+0000","last_became_peered":"2024-04-26T20:32:26.241783+0000","last_unstale":"2024-04-26T20:33:30.393346+0000","last_undegraded":"2024-04-26T20:33:30.393346+0000","last_fullsized":"2024-04-26T20:33:30.393346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:20.821967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384804+0000","last_change":"2024-04-26T20:32:30.382937+0000","last_active":"2024-04-26T20:33:30.384804+0000","last_peered":"2024-04-26T20:33:30.384804+0000","last_clean":"2024-04-26T20:33:30.384804+0000","last_became_active":"2024-04-26T20:32:30.378828+0000","last_became_peered":"2024-04-26T20:32:30.378828+0000","last_unstale":"2024-04-26T20:33:30.384804+0000","last_undegraded":"2024-04-26T20:33:30.384804+0000","last_fullsized":"2024-04-26T20:33:30.384804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:33:41.685012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":195,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:16.830304+0000","last_change":"2024-04-26T20:32:25.280362+0000","last_active":"2024-04-26T20:34:16.830304+0000","last_peered":"2024-04-26T20:34:16.830304+0000","last_clean":"2024-04-26T20:34:16.830304+0000","last_became_active":"2024-04-26T20:32:25.279975+0000","last_became_peered":"2024-04-26T20:32:25.279975+0000","last_unstale":"2024-04-26T20:34:16.830304+0000","last_undegraded":"2024-04-26T20:34:16.830304+0000","last_fullsized":"2024-04-26T20:34:16.830304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:08.150635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":164,"num_read_kb":164,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.411093+0000","last_change":"2024-04-26T20:32:26.231231+0000","last_active":"2024-04-26T20:33:30.411093+0000","last_peered":"2024-04-26T20:33:30.411093+0000","last_clean":"2024-04-26T20:33:30.411093+0000","last_became_active":"2024-04-26T20:32:26.230831+0000","last_became_peered":"2024-04-26T20:32:26.230831+0000","last_unstale":"2024-04-26T20:33:30.411093+0000","last_undegraded":"2024-04-26T20:33:30.411093+0000","last_fullsized":"2024-04-26T20:33:30.411093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:03:45.272542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.401129+0000","last_change":"2024-04-26T20:32:30.382633+0000","last_active":"2024-04-26T20:33:30.401129+0000","last_peered":"2024-04-26T20:33:30.401129+0000","last_clean":"2024-04-26T20:33:30.401129+0000","last_became_active":"2024-04-26T20:32:30.379821+0000","last_became_peered":"2024-04-26T20:32:30.379821+0000","last_unstale":"2024-04-26T20:33:30.401129+0000","last_undegraded":"2024-04-26T20:33:30.401129+0000","last_fullsized":"2024-04-26T20:33:30.401129+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:58:16.888172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395693+0000","last_change":"2024-04-26T20:32:25.264844+0000","last_active":"2024-04-26T20:33:30.395693+0000","last_peered":"2024-04-26T20:33:30.395693+0000","last_clean":"2024-04-26T20:33:30.395693+0000","last_became_active":"2024-04-26T20:32:25.264097+0000","last_became_peered":"2024-04-26T20:32:25.264097+0000","last_unstale":"2024-04-26T20:33:30.395693+0000","last_undegraded":"2024-04-26T20:33:30.395693+0000","last_fullsized":"2024-04-26T20:33:30.395693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:09.308656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.400892+0000","last_change":"2024-04-26T20:33:30.400892+0000","last_active":"2024-04-26T20:33:30.400892+0000","last_peered":"2024-04-26T20:33:30.400892+0000","last_clean":"2024-04-26T20:33:30.400892+0000","last_became_active":"2024-04-26T20:33:30.398674+0000","last_became_peered":"2024-04-26T20:33:30.398674+0000","last_unstale":"2024-04-26T20:33:30.400892+0000","last_undegraded":"2024-04-26T20:33:30.400892+0000","last_fullsized":"2024-04-26T20:33:30.400892+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:03:08.573433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.412074+0000","last_change":"2024-04-26T20:33:30.412074+0000","last_active":"2024-04-26T20:33:30.412074+0000","last_peered":"2024-04-26T20:33:30.412074+0000","last_clean":"2024-04-26T20:33:30.412074+0000","last_became_active":"2024-04-26T20:33:30.411535+0000","last_became_peered":"2024-04-26T20:33:30.411535+0000","last_unstale":"2024-04-26T20:33:30.412074+0000","last_undegraded":"2024-04-26T20:33:30.412074+0000","last_fullsized":"2024-04-26T20:33:30.412074+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:54:52.809109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,10],"acting":[4,1,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391704+0000","last_change":"2024-04-26T20:32:25.284996+0000","last_active":"2024-04-26T20:33:30.391704+0000","last_peered":"2024-04-26T20:33:30.391704+0000","last_clean":"2024-04-26T20:33:30.391704+0000","last_became_active":"2024-04-26T20:32:25.284702+0000","last_became_peered":"2024-04-26T20:32:25.284702+0000","last_unstale":"2024-04-26T20:33:30.391704+0000","last_undegraded":"2024-04-26T20:33:30.391704+0000","last_fullsized":"2024-04-26T20:33:30.391704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:39.831522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410826+0000","last_change":"2024-04-26T20:32:26.230357+0000","last_active":"2024-04-26T20:33:30.410826+0000","last_peered":"2024-04-26T20:33:30.410826+0000","last_clean":"2024-04-26T20:33:30.410826+0000","last_became_active":"2024-04-26T20:32:26.229831+0000","last_became_peered":"2024-04-26T20:32:26.229831+0000","last_unstale":"2024-04-26T20:33:30.410826+0000","last_undegraded":"2024-04-26T20:33:30.410826+0000","last_fullsized":"2024-04-26T20:33:30.410826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:20:12.452768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.401654+0000","last_change":"2024-04-26T20:32:30.383469+0000","last_active":"2024-04-26T20:33:30.401654+0000","last_peered":"2024-04-26T20:33:30.401654+0000","last_clean":"2024-04-26T20:33:30.401654+0000","last_became_active":"2024-04-26T20:32:30.382416+0000","last_became_peered":"2024-04-26T20:32:30.382416+0000","last_unstale":"2024-04-26T20:33:30.401654+0000","last_undegraded":"2024-04-26T20:33:30.401654+0000","last_fullsized":"2024-04-26T20:33:30.401654+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:55.254214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":231,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:17.460831+0000","last_change":"2024-04-26T20:32:25.264433+0000","last_active":"2024-04-26T20:34:17.460831+0000","last_peered":"2024-04-26T20:34:17.460831+0000","last_clean":"2024-04-26T20:34:17.460831+0000","last_became_active":"2024-04-26T20:32:25.264104+0000","last_became_peered":"2024-04-26T20:32:25.264104+0000","last_unstale":"2024-04-26T20:34:17.460831+0000","last_undegraded":"2024-04-26T20:34:17.460831+0000","last_fullsized":"2024-04-26T20:34:17.460831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.030082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54826,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":169,"num_read_kb":137,"num_write":37,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383114+0000","last_change":"2024-04-26T20:32:26.219704+0000","last_active":"2024-04-26T20:33:30.383114+0000","last_peered":"2024-04-26T20:33:30.383114+0000","last_clean":"2024-04-26T20:33:30.383114+0000","last_became_active":"2024-04-26T20:32:26.219397+0000","last_became_peered":"2024-04-26T20:32:26.219397+0000","last_unstale":"2024-04-26T20:33:30.383114+0000","last_undegraded":"2024-04-26T20:33:30.383114+0000","last_fullsized":"2024-04-26T20:33:30.383114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:04.570044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.392486+0000","last_change":"2024-04-26T20:32:30.388828+0000","last_active":"2024-04-26T20:33:30.392486+0000","last_peered":"2024-04-26T20:33:30.392486+0000","last_clean":"2024-04-26T20:33:30.392486+0000","last_became_active":"2024-04-26T20:32:30.387748+0000","last_became_peered":"2024-04-26T20:32:30.387748+0000","last_unstale":"2024-04-26T20:33:30.392486+0000","last_undegraded":"2024-04-26T20:33:30.392486+0000","last_fullsized":"2024-04-26T20:33:30.392486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:53:09.900151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391784+0000","last_change":"2024-04-26T20:32:25.271910+0000","last_active":"2024-04-26T20:33:30.391784+0000","last_peered":"2024-04-26T20:33:30.391784+0000","last_clean":"2024-04-26T20:33:30.391784+0000","last_became_active":"2024-04-26T20:32:25.271572+0000","last_became_peered":"2024-04-26T20:32:25.271572+0000","last_unstale":"2024-04-26T20:33:30.391784+0000","last_undegraded":"2024-04-26T20:33:30.391784+0000","last_fullsized":"2024-04-26T20:33:30.391784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:32:59.013229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395265+0000","last_change":"2024-04-26T20:32:26.237691+0000","last_active":"2024-04-26T20:33:30.395265+0000","last_peered":"2024-04-26T20:33:30.395265+0000","last_clean":"2024-04-26T20:33:30.395265+0000","last_became_active":"2024-04-26T20:32:26.237338+0000","last_became_peered":"2024-04-26T20:32:26.237338+0000","last_unstale":"2024-04-26T20:33:30.395265+0000","last_undegraded":"2024-04-26T20:33:30.395265+0000","last_fullsized":"2024-04-26T20:33:30.395265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:51.782689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385493+0000","last_change":"2024-04-26T20:32:30.388485+0000","last_active":"2024-04-26T20:33:30.385493+0000","last_peered":"2024-04-26T20:33:30.385493+0000","last_clean":"2024-04-26T20:33:30.385493+0000","last_became_active":"2024-04-26T20:32:30.388074+0000","last_became_peered":"2024-04-26T20:32:30.388074+0000","last_unstale":"2024-04-26T20:33:30.385493+0000","last_undegraded":"2024-04-26T20:33:30.385493+0000","last_fullsized":"2024-04-26T20:33:30.385493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:52.792551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391873+0000","last_change":"2024-04-26T20:32:25.271589+0000","last_active":"2024-04-26T20:33:30.391873+0000","last_peered":"2024-04-26T20:33:30.391873+0000","last_clean":"2024-04-26T20:33:30.391873+0000","last_became_active":"2024-04-26T20:32:25.271260+0000","last_became_peered":"2024-04-26T20:32:25.271260+0000","last_unstale":"2024-04-26T20:33:30.391873+0000","last_undegraded":"2024-04-26T20:33:30.391873+0000","last_fullsized":"2024-04-26T20:33:30.391873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:30.978561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395338+0000","last_change":"2024-04-26T20:32:26.240158+0000","last_active":"2024-04-26T20:33:30.395338+0000","last_peered":"2024-04-26T20:33:30.395338+0000","last_clean":"2024-04-26T20:33:30.395338+0000","last_became_active":"2024-04-26T20:32:26.239831+0000","last_became_peered":"2024-04-26T20:32:26.239831+0000","last_unstale":"2024-04-26T20:33:30.395338+0000","last_undegraded":"2024-04-26T20:33:30.395338+0000","last_fullsized":"2024-04-26T20:33:30.395338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:07.870097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.401518+0000","last_change":"2024-04-26T20:32:30.392090+0000","last_active":"2024-04-26T20:33:30.401518+0000","last_peered":"2024-04-26T20:33:30.401518+0000","last_clean":"2024-04-26T20:33:30.401518+0000","last_became_active":"2024-04-26T20:32:30.390404+0000","last_became_peered":"2024-04-26T20:32:30.390404+0000","last_unstale":"2024-04-26T20:33:30.401518+0000","last_undegraded":"2024-04-26T20:33:30.401518+0000","last_fullsized":"2024-04-26T20:33:30.401518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:50:10.743133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:17.887759+0000","last_change":"2024-04-26T20:32:25.264291+0000","last_active":"2024-04-26T20:34:17.887759+0000","last_peered":"2024-04-26T20:34:17.887759+0000","last_clean":"2024-04-26T20:34:17.887759+0000","last_became_active":"2024-04-26T20:32:25.263901+0000","last_became_peered":"2024-04-26T20:32:25.263901+0000","last_unstale":"2024-04-26T20:34:17.887759+0000","last_undegraded":"2024-04-26T20:34:17.887759+0000","last_fullsized":"2024-04-26T20:34:17.887759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:23:28.851941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395208+0000","last_change":"2024-04-26T20:32:26.221472+0000","last_active":"2024-04-26T20:33:30.395208+0000","last_peered":"2024-04-26T20:33:30.395208+0000","last_clean":"2024-04-26T20:33:30.395208+0000","last_became_active":"2024-04-26T20:32:26.221117+0000","last_became_peered":"2024-04-26T20:32:26.221117+0000","last_unstale":"2024-04-26T20:33:30.395208+0000","last_undegraded":"2024-04-26T20:33:30.395208+0000","last_fullsized":"2024-04-26T20:33:30.395208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:27:01.666749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.400851+0000","last_change":"2024-04-26T20:32:30.392000+0000","last_active":"2024-04-26T20:33:30.400851+0000","last_peered":"2024-04-26T20:33:30.400851+0000","last_clean":"2024-04-26T20:33:30.400851+0000","last_became_active":"2024-04-26T20:32:30.386933+0000","last_became_peered":"2024-04-26T20:32:30.386933+0000","last_unstale":"2024-04-26T20:33:30.400851+0000","last_undegraded":"2024-04-26T20:33:30.400851+0000","last_fullsized":"2024-04-26T20:33:30.400851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:23.666439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382595+0000","last_change":"2024-04-26T20:32:25.276482+0000","last_active":"2024-04-26T20:33:30.382595+0000","last_peered":"2024-04-26T20:33:30.382595+0000","last_clean":"2024-04-26T20:33:30.382595+0000","last_became_active":"2024-04-26T20:32:25.276159+0000","last_became_peered":"2024-04-26T20:32:25.276159+0000","last_unstale":"2024-04-26T20:33:30.382595+0000","last_undegraded":"2024-04-26T20:33:30.382595+0000","last_fullsized":"2024-04-26T20:33:30.382595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:34.029524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393349+0000","last_change":"2024-04-26T20:32:26.225344+0000","last_active":"2024-04-26T20:33:30.393349+0000","last_peered":"2024-04-26T20:33:30.393349+0000","last_clean":"2024-04-26T20:33:30.393349+0000","last_became_active":"2024-04-26T20:32:26.225039+0000","last_became_peered":"2024-04-26T20:32:26.225039+0000","last_unstale":"2024-04-26T20:33:30.393349+0000","last_undegraded":"2024-04-26T20:33:30.393349+0000","last_fullsized":"2024-04-26T20:33:30.393349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:08.680765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393327+0000","last_change":"2024-04-26T20:32:30.389274+0000","last_active":"2024-04-26T20:33:30.393327+0000","last_peered":"2024-04-26T20:33:30.393327+0000","last_clean":"2024-04-26T20:33:30.393327+0000","last_became_active":"2024-04-26T20:32:30.388527+0000","last_became_peered":"2024-04-26T20:32:30.388527+0000","last_unstale":"2024-04-26T20:33:30.393327+0000","last_undegraded":"2024-04-26T20:33:30.393327+0000","last_fullsized":"2024-04-26T20:33:30.393327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:14.689223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383283+0000","last_change":"2024-04-26T20:32:25.267338+0000","last_active":"2024-04-26T20:33:30.383283+0000","last_peered":"2024-04-26T20:33:30.383283+0000","last_clean":"2024-04-26T20:33:30.383283+0000","last_became_active":"2024-04-26T20:32:25.266234+0000","last_became_peered":"2024-04-26T20:32:25.266234+0000","last_unstale":"2024-04-26T20:33:30.383283+0000","last_undegraded":"2024-04-26T20:33:30.383283+0000","last_fullsized":"2024-04-26T20:33:30.383283+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:55.603147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384931+0000","last_change":"2024-04-26T20:32:26.222686+0000","last_active":"2024-04-26T20:33:30.384931+0000","last_peered":"2024-04-26T20:33:30.384931+0000","last_clean":"2024-04-26T20:33:30.384931+0000","last_became_active":"2024-04-26T20:32:26.222364+0000","last_became_peered":"2024-04-26T20:32:26.222364+0000","last_unstale":"2024-04-26T20:33:30.384931+0000","last_undegraded":"2024-04-26T20:33:30.384931+0000","last_fullsized":"2024-04-26T20:33:30.384931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:56.170913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382708+0000","last_change":"2024-04-26T20:32:30.414367+0000","last_active":"2024-04-26T20:33:30.382708+0000","last_peered":"2024-04-26T20:33:30.382708+0000","last_clean":"2024-04-26T20:33:30.382708+0000","last_became_active":"2024-04-26T20:32:30.413889+0000","last_became_peered":"2024-04-26T20:32:30.413889+0000","last_unstale":"2024-04-26T20:33:30.382708+0000","last_undegraded":"2024-04-26T20:33:30.382708+0000","last_fullsized":"2024-04-26T20:33:30.382708+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:30:11.336500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.402415+0000","last_change":"2024-04-26T20:32:25.286408+0000","last_active":"2024-04-26T20:33:30.402415+0000","last_peered":"2024-04-26T20:33:30.402415+0000","last_clean":"2024-04-26T20:33:30.402415+0000","last_became_active":"2024-04-26T20:32:25.285986+0000","last_became_peered":"2024-04-26T20:32:25.285986+0000","last_unstale":"2024-04-26T20:33:30.402415+0000","last_undegraded":"2024-04-26T20:33:30.402415+0000","last_fullsized":"2024-04-26T20:33:30.402415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:47.757720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.386184+0000","last_change":"2024-04-26T20:32:26.250459+0000","last_active":"2024-04-26T20:33:30.386184+0000","last_peered":"2024-04-26T20:33:30.386184+0000","last_clean":"2024-04-26T20:33:30.386184+0000","last_became_active":"2024-04-26T20:32:26.250154+0000","last_became_peered":"2024-04-26T20:32:26.250154+0000","last_unstale":"2024-04-26T20:33:30.386184+0000","last_undegraded":"2024-04-26T20:33:30.386184+0000","last_fullsized":"2024-04-26T20:33:30.386184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:47.185067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.398483+0000","last_change":"2024-04-26T20:32:30.395258+0000","last_active":"2024-04-26T20:33:30.398483+0000","last_peered":"2024-04-26T20:33:30.398483+0000","last_clean":"2024-04-26T20:33:30.398483+0000","last_became_active":"2024-04-26T20:32:30.392452+0000","last_became_peered":"2024-04-26T20:32:30.392452+0000","last_unstale":"2024-04-26T20:33:30.398483+0000","last_undegraded":"2024-04-26T20:33:30.398483+0000","last_fullsized":"2024-04-26T20:33:30.398483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:19.883901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:16.885302+0000","last_change":"2024-04-26T20:32:25.297361+0000","last_active":"2024-04-26T20:34:16.885302+0000","last_peered":"2024-04-26T20:34:16.885302+0000","last_clean":"2024-04-26T20:34:16.885302+0000","last_became_active":"2024-04-26T20:32:25.297125+0000","last_became_peered":"2024-04-26T20:32:25.297125+0000","last_unstale":"2024-04-26T20:34:16.885302+0000","last_undegraded":"2024-04-26T20:34:16.885302+0000","last_fullsized":"2024-04-26T20:34:16.885302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:14:38.038683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385946+0000","last_change":"2024-04-26T20:32:26.249371+0000","last_active":"2024-04-26T20:33:30.385946+0000","last_peered":"2024-04-26T20:33:30.385946+0000","last_clean":"2024-04-26T20:33:30.385946+0000","last_became_active":"2024-04-26T20:32:26.249018+0000","last_became_peered":"2024-04-26T20:32:26.249018+0000","last_unstale":"2024-04-26T20:33:30.385946+0000","last_undegraded":"2024-04-26T20:33:30.385946+0000","last_fullsized":"2024-04-26T20:33:30.385946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:27.553928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.401076+0000","last_change":"2024-04-26T20:32:30.414854+0000","last_active":"2024-04-26T20:33:30.401076+0000","last_peered":"2024-04-26T20:33:30.401076+0000","last_clean":"2024-04-26T20:33:30.401076+0000","last_became_active":"2024-04-26T20:32:30.414374+0000","last_became_peered":"2024-04-26T20:32:30.414374+0000","last_unstale":"2024-04-26T20:33:30.401076+0000","last_undegraded":"2024-04-26T20:33:30.401076+0000","last_fullsized":"2024-04-26T20:33:30.401076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:32.071225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391794+0000","last_change":"2024-04-26T20:32:26.222681+0000","last_active":"2024-04-26T20:33:30.391794+0000","last_peered":"2024-04-26T20:33:30.391794+0000","last_clean":"2024-04-26T20:33:30.391794+0000","last_became_active":"2024-04-26T20:32:26.222099+0000","last_became_peered":"2024-04-26T20:32:26.222099+0000","last_unstale":"2024-04-26T20:33:30.391794+0000","last_undegraded":"2024-04-26T20:33:30.391794+0000","last_fullsized":"2024-04-26T20:33:30.391794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:35:21.958180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:16.884424+0000","last_change":"2024-04-26T20:32:25.276314+0000","last_active":"2024-04-26T20:34:16.884424+0000","last_peered":"2024-04-26T20:34:16.884424+0000","last_clean":"2024-04-26T20:34:16.884424+0000","last_became_active":"2024-04-26T20:32:25.275926+0000","last_became_peered":"2024-04-26T20:32:25.275926+0000","last_unstale":"2024-04-26T20:34:16.884424+0000","last_undegraded":"2024-04-26T20:34:16.884424+0000","last_fullsized":"2024-04-26T20:34:16.884424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:25.027777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395299+0000","last_change":"2024-04-26T20:32:30.385121+0000","last_active":"2024-04-26T20:33:30.395299+0000","last_peered":"2024-04-26T20:33:30.395299+0000","last_clean":"2024-04-26T20:33:30.395299+0000","last_became_active":"2024-04-26T20:32:30.384697+0000","last_became_peered":"2024-04-26T20:32:30.384697+0000","last_unstale":"2024-04-26T20:33:30.395299+0000","last_undegraded":"2024-04-26T20:33:30.395299+0000","last_fullsized":"2024-04-26T20:33:30.395299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:00.725130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385691+0000","last_change":"2024-04-26T20:32:26.227550+0000","last_active":"2024-04-26T20:33:30.385691+0000","last_peered":"2024-04-26T20:33:30.385691+0000","last_clean":"2024-04-26T20:33:30.385691+0000","last_became_active":"2024-04-26T20:32:26.224262+0000","last_became_peered":"2024-04-26T20:32:26.224262+0000","last_unstale":"2024-04-26T20:33:30.385691+0000","last_undegraded":"2024-04-26T20:33:30.385691+0000","last_fullsized":"2024-04-26T20:33:30.385691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:01.413870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383169+0000","last_change":"2024-04-26T20:32:25.274253+0000","last_active":"2024-04-26T20:33:30.383169+0000","last_peered":"2024-04-26T20:33:30.383169+0000","last_clean":"2024-04-26T20:33:30.383169+0000","last_became_active":"2024-04-26T20:32:25.273956+0000","last_became_peered":"2024-04-26T20:32:25.273956+0000","last_unstale":"2024-04-26T20:33:30.383169+0000","last_undegraded":"2024-04-26T20:33:30.383169+0000","last_fullsized":"2024-04-26T20:33:30.383169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:23:27.832135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":353,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395071+0000","last_change":"2024-04-26T20:31:05.644347+0000","last_active":"2024-04-26T20:33:30.395071+0000","last_peered":"2024-04-26T20:33:30.395071+0000","last_clean":"2024-04-26T20:33:30.395071+0000","last_became_active":"2024-04-26T20:31:05.643888+0000","last_became_peered":"2024-04-26T20:31:05.643888+0000","last_unstale":"2024-04-26T20:33:30.395071+0000","last_undegraded":"2024-04-26T20:33:30.395071+0000","last_fullsized":"2024-04-26T20:33:30.395071+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:27:11.501328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:27:11.501328+0000","last_clean_scrub_stamp":"2024-04-26T20:27:11.501328+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:08.802687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.406111+0000","last_change":"2024-04-26T20:33:30.406111+0000","last_active":"2024-04-26T20:33:30.406111+0000","last_peered":"2024-04-26T20:33:30.406111+0000","last_clean":"2024-04-26T20:33:30.406111+0000","last_became_active":"2024-04-26T20:33:30.405639+0000","last_became_peered":"2024-04-26T20:33:30.405639+0000","last_unstale":"2024-04-26T20:33:30.406111+0000","last_undegraded":"2024-04-26T20:33:30.406111+0000","last_fullsized":"2024-04-26T20:33:30.406111+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:15:50.668699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.396839+0000","last_change":"2024-04-26T20:32:26.239936+0000","last_active":"2024-04-26T20:33:30.396839+0000","last_peered":"2024-04-26T20:33:30.396839+0000","last_clean":"2024-04-26T20:33:30.396839+0000","last_became_active":"2024-04-26T20:32:26.239583+0000","last_became_peered":"2024-04-26T20:32:26.239583+0000","last_unstale":"2024-04-26T20:33:30.396839+0000","last_undegraded":"2024-04-26T20:33:30.396839+0000","last_fullsized":"2024-04-26T20:33:30.396839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:57.183460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395107+0000","last_change":"2024-04-26T20:32:25.267876+0000","last_active":"2024-04-26T20:33:30.395107+0000","last_peered":"2024-04-26T20:33:30.395107+0000","last_clean":"2024-04-26T20:33:30.395107+0000","last_became_active":"2024-04-26T20:32:25.267573+0000","last_became_peered":"2024-04-26T20:32:25.267573+0000","last_unstale":"2024-04-26T20:33:30.395107+0000","last_undegraded":"2024-04-26T20:33:30.395107+0000","last_fullsized":"2024-04-26T20:33:30.395107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:43:55.630426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384980+0000","last_change":"2024-04-26T20:32:30.383364+0000","last_active":"2024-04-26T20:33:30.384980+0000","last_peered":"2024-04-26T20:33:30.384980+0000","last_clean":"2024-04-26T20:33:30.384980+0000","last_became_active":"2024-04-26T20:32:30.382875+0000","last_became_peered":"2024-04-26T20:32:30.382875+0000","last_unstale":"2024-04-26T20:33:30.384980+0000","last_undegraded":"2024-04-26T20:33:30.384980+0000","last_fullsized":"2024-04-26T20:33:30.384980+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:55.112888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383125+0000","last_change":"2024-04-26T20:32:25.280508+0000","last_active":"2024-04-26T20:33:30.383125+0000","last_peered":"2024-04-26T20:33:30.383125+0000","last_clean":"2024-04-26T20:33:30.383125+0000","last_became_active":"2024-04-26T20:32:25.280265+0000","last_became_peered":"2024-04-26T20:32:25.280265+0000","last_unstale":"2024-04-26T20:33:30.383125+0000","last_undegraded":"2024-04-26T20:33:30.383125+0000","last_fullsized":"2024-04-26T20:33:30.383125+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:16:50.427659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.398818+0000","last_change":"2024-04-26T20:32:26.255230+0000","last_active":"2024-04-26T20:33:30.398818+0000","last_peered":"2024-04-26T20:33:30.398818+0000","last_clean":"2024-04-26T20:33:30.398818+0000","last_became_active":"2024-04-26T20:32:26.254099+0000","last_became_peered":"2024-04-26T20:32:26.254099+0000","last_unstale":"2024-04-26T20:33:30.398818+0000","last_undegraded":"2024-04-26T20:33:30.398818+0000","last_fullsized":"2024-04-26T20:33:30.398818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:41.895816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384464+0000","last_change":"2024-04-26T20:32:30.414688+0000","last_active":"2024-04-26T20:33:30.384464+0000","last_peered":"2024-04-26T20:33:30.384464+0000","last_clean":"2024-04-26T20:33:30.384464+0000","last_became_active":"2024-04-26T20:32:30.414052+0000","last_became_peered":"2024-04-26T20:32:30.414052+0000","last_unstale":"2024-04-26T20:33:30.384464+0000","last_undegraded":"2024-04-26T20:33:30.384464+0000","last_fullsized":"2024-04-26T20:33:30.384464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:24.943559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.403192+0000","last_change":"2024-04-26T20:32:25.298609+0000","last_active":"2024-04-26T20:33:30.403192+0000","last_peered":"2024-04-26T20:33:30.403192+0000","last_clean":"2024-04-26T20:33:30.403192+0000","last_became_active":"2024-04-26T20:32:25.298155+0000","last_became_peered":"2024-04-26T20:32:25.298155+0000","last_unstale":"2024-04-26T20:33:30.403192+0000","last_undegraded":"2024-04-26T20:33:30.403192+0000","last_fullsized":"2024-04-26T20:33:30.403192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:37.130994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.388963+0000","last_change":"2024-04-26T20:32:26.231203+0000","last_active":"2024-04-26T20:33:30.388963+0000","last_peered":"2024-04-26T20:33:30.388963+0000","last_clean":"2024-04-26T20:33:30.388963+0000","last_became_active":"2024-04-26T20:32:26.230775+0000","last_became_peered":"2024-04-26T20:32:26.230775+0000","last_unstale":"2024-04-26T20:33:30.388963+0000","last_undegraded":"2024-04-26T20:33:30.388963+0000","last_fullsized":"2024-04-26T20:33:30.388963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:40.557744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391393+0000","last_change":"2024-04-26T20:32:30.375703+0000","last_active":"2024-04-26T20:33:30.391393+0000","last_peered":"2024-04-26T20:33:30.391393+0000","last_clean":"2024-04-26T20:33:30.391393+0000","last_became_active":"2024-04-26T20:32:30.375292+0000","last_became_peered":"2024-04-26T20:32:30.375292+0000","last_unstale":"2024-04-26T20:33:30.391393+0000","last_undegraded":"2024-04-26T20:33:30.391393+0000","last_fullsized":"2024-04-26T20:33:30.391393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:39.747426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383571+0000","last_change":"2024-04-26T20:32:25.290908+0000","last_active":"2024-04-26T20:33:30.383571+0000","last_peered":"2024-04-26T20:33:30.383571+0000","last_clean":"2024-04-26T20:33:30.383571+0000","last_became_active":"2024-04-26T20:32:25.290655+0000","last_became_peered":"2024-04-26T20:32:25.290655+0000","last_unstale":"2024-04-26T20:33:30.383571+0000","last_undegraded":"2024-04-26T20:33:30.383571+0000","last_fullsized":"2024-04-26T20:33:30.383571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:28:42.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394820+0000","last_change":"2024-04-26T20:32:26.246554+0000","last_active":"2024-04-26T20:33:30.394820+0000","last_peered":"2024-04-26T20:33:30.394820+0000","last_clean":"2024-04-26T20:33:30.394820+0000","last_became_active":"2024-04-26T20:32:26.246279+0000","last_became_peered":"2024-04-26T20:32:26.246279+0000","last_unstale":"2024-04-26T20:33:30.394820+0000","last_undegraded":"2024-04-26T20:33:30.394820+0000","last_fullsized":"2024-04-26T20:33:30.394820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:02.561410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383250+0000","last_change":"2024-04-26T20:32:30.414622+0000","last_active":"2024-04-26T20:33:30.383250+0000","last_peered":"2024-04-26T20:33:30.383250+0000","last_clean":"2024-04-26T20:33:30.383250+0000","last_became_active":"2024-04-26T20:32:30.414176+0000","last_became_peered":"2024-04-26T20:32:30.414176+0000","last_unstale":"2024-04-26T20:33:30.383250+0000","last_undegraded":"2024-04-26T20:33:30.383250+0000","last_fullsized":"2024-04-26T20:33:30.383250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:15.011512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410491+0000","last_change":"2024-04-26T20:32:25.275220+0000","last_active":"2024-04-26T20:33:30.410491+0000","last_peered":"2024-04-26T20:33:30.410491+0000","last_clean":"2024-04-26T20:33:30.410491+0000","last_became_active":"2024-04-26T20:32:25.274990+0000","last_became_peered":"2024-04-26T20:32:25.274990+0000","last_unstale":"2024-04-26T20:33:30.410491+0000","last_undegraded":"2024-04-26T20:33:30.410491+0000","last_fullsized":"2024-04-26T20:33:30.410491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:58.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382593+0000","last_change":"2024-04-26T20:32:26.218481+0000","last_active":"2024-04-26T20:33:30.382593+0000","last_peered":"2024-04-26T20:33:30.382593+0000","last_clean":"2024-04-26T20:33:30.382593+0000","last_became_active":"2024-04-26T20:32:26.218146+0000","last_became_peered":"2024-04-26T20:32:26.218146+0000","last_unstale":"2024-04-26T20:33:30.382593+0000","last_undegraded":"2024-04-26T20:33:30.382593+0000","last_fullsized":"2024-04-26T20:33:30.382593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:49:37.310945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.401948+0000","last_change":"2024-04-26T20:32:30.383815+0000","last_active":"2024-04-26T20:33:30.401948+0000","last_peered":"2024-04-26T20:33:30.401948+0000","last_clean":"2024-04-26T20:33:30.401948+0000","last_became_active":"2024-04-26T20:32:30.382818+0000","last_became_peered":"2024-04-26T20:32:30.382818+0000","last_unstale":"2024-04-26T20:33:30.401948+0000","last_undegraded":"2024-04-26T20:33:30.401948+0000","last_fullsized":"2024-04-26T20:33:30.401948+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:47.839781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385330+0000","last_change":"2024-04-26T20:32:25.291527+0000","last_active":"2024-04-26T20:33:30.385330+0000","last_peered":"2024-04-26T20:33:30.385330+0000","last_clean":"2024-04-26T20:33:30.385330+0000","last_became_active":"2024-04-26T20:32:25.285353+0000","last_became_peered":"2024-04-26T20:32:25.285353+0000","last_unstale":"2024-04-26T20:33:30.385330+0000","last_undegraded":"2024-04-26T20:33:30.385330+0000","last_fullsized":"2024-04-26T20:33:30.385330+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:28.927822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384707+0000","last_change":"2024-04-26T20:32:26.241064+0000","last_active":"2024-04-26T20:33:30.384707+0000","last_peered":"2024-04-26T20:33:30.384707+0000","last_clean":"2024-04-26T20:33:30.384707+0000","last_became_active":"2024-04-26T20:32:26.240673+0000","last_became_peered":"2024-04-26T20:32:26.240673+0000","last_unstale":"2024-04-26T20:33:30.384707+0000","last_undegraded":"2024-04-26T20:33:30.384707+0000","last_fullsized":"2024-04-26T20:33:30.384707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:19:47.777751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383143+0000","last_change":"2024-04-26T20:32:30.388465+0000","last_active":"2024-04-26T20:33:30.383143+0000","last_peered":"2024-04-26T20:33:30.383143+0000","last_clean":"2024-04-26T20:33:30.383143+0000","last_became_active":"2024-04-26T20:32:30.388038+0000","last_became_peered":"2024-04-26T20:32:30.388038+0000","last_unstale":"2024-04-26T20:33:30.383143+0000","last_undegraded":"2024-04-26T20:33:30.383143+0000","last_fullsized":"2024-04-26T20:33:30.383143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:38:39.596970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.392793+0000","last_change":"2024-04-26T20:32:25.271914+0000","last_active":"2024-04-26T20:33:30.392793+0000","last_peered":"2024-04-26T20:33:30.392793+0000","last_clean":"2024-04-26T20:33:30.392793+0000","last_became_active":"2024-04-26T20:32:25.271567+0000","last_became_peered":"2024-04-26T20:32:25.271567+0000","last_unstale":"2024-04-26T20:33:30.392793+0000","last_undegraded":"2024-04-26T20:33:30.392793+0000","last_fullsized":"2024-04-26T20:33:30.392793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:02.182470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385349+0000","last_change":"2024-04-26T20:32:26.245344+0000","last_active":"2024-04-26T20:33:30.385349+0000","last_peered":"2024-04-26T20:33:30.385349+0000","last_clean":"2024-04-26T20:33:30.385349+0000","last_became_active":"2024-04-26T20:32:26.245032+0000","last_became_peered":"2024-04-26T20:32:26.245032+0000","last_unstale":"2024-04-26T20:33:30.385349+0000","last_undegraded":"2024-04-26T20:33:30.385349+0000","last_fullsized":"2024-04-26T20:33:30.385349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:12:51.368935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.383166+0000","last_change":"2024-04-26T20:32:30.414408+0000","last_active":"2024-04-26T20:33:30.383166+0000","last_peered":"2024-04-26T20:33:30.383166+0000","last_clean":"2024-04-26T20:33:30.383166+0000","last_became_active":"2024-04-26T20:32:30.413892+0000","last_became_peered":"2024-04-26T20:32:30.413892+0000","last_unstale":"2024-04-26T20:33:30.383166+0000","last_undegraded":"2024-04-26T20:33:30.383166+0000","last_fullsized":"2024-04-26T20:33:30.383166+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:53.332840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.382766+0000","last_change":"2024-04-26T20:32:25.265440+0000","last_active":"2024-04-26T20:33:30.382766+0000","last_peered":"2024-04-26T20:33:30.382766+0000","last_clean":"2024-04-26T20:33:30.382766+0000","last_became_active":"2024-04-26T20:32:25.265129+0000","last_became_peered":"2024-04-26T20:32:25.265129+0000","last_unstale":"2024-04-26T20:33:30.382766+0000","last_undegraded":"2024-04-26T20:33:30.382766+0000","last_fullsized":"2024-04-26T20:33:30.382766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:49.475009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395650+0000","last_change":"2024-04-26T20:32:26.234477+0000","last_active":"2024-04-26T20:33:30.395650+0000","last_peered":"2024-04-26T20:33:30.395650+0000","last_clean":"2024-04-26T20:33:30.395650+0000","last_became_active":"2024-04-26T20:32:26.234103+0000","last_became_peered":"2024-04-26T20:32:26.234103+0000","last_unstale":"2024-04-26T20:33:30.395650+0000","last_undegraded":"2024-04-26T20:33:30.395650+0000","last_fullsized":"2024-04-26T20:33:30.395650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:54.098860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410477+0000","last_change":"2024-04-26T20:32:30.391466+0000","last_active":"2024-04-26T20:33:30.410477+0000","last_peered":"2024-04-26T20:33:30.410477+0000","last_clean":"2024-04-26T20:33:30.410477+0000","last_became_active":"2024-04-26T20:32:30.391037+0000","last_became_peered":"2024-04-26T20:32:30.391037+0000","last_unstale":"2024-04-26T20:33:30.410477+0000","last_undegraded":"2024-04-26T20:33:30.410477+0000","last_fullsized":"2024-04-26T20:33:30.410477+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T20:39:16.305520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":120,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:34:19.835482+0000","last_change":"2024-04-26T20:32:25.281688+0000","last_active":"2024-04-26T20:34:19.835482+0000","last_peered":"2024-04-26T20:34:19.835482+0000","last_clean":"2024-04-26T20:34:19.835482+0000","last_became_active":"2024-04-26T20:32:25.281430+0000","last_became_peered":"2024-04-26T20:32:25.281430+0000","last_unstale":"2024-04-26T20:34:19.835482+0000","last_undegraded":"2024-04-26T20:34:19.835482+0000","last_fullsized":"2024-04-26T20:34:19.835482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:52.785532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395661+0000","last_change":"2024-04-26T20:32:26.226468+0000","last_active":"2024-04-26T20:33:30.395661+0000","last_peered":"2024-04-26T20:33:30.395661+0000","last_clean":"2024-04-26T20:33:30.395661+0000","last_became_active":"2024-04-26T20:32:26.225399+0000","last_became_peered":"2024-04-26T20:32:26.225399+0000","last_unstale":"2024-04-26T20:33:30.395661+0000","last_undegraded":"2024-04-26T20:33:30.395661+0000","last_fullsized":"2024-04-26T20:33:30.395661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:17:11.787658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.393396+0000","last_change":"2024-04-26T20:32:30.383646+0000","last_active":"2024-04-26T20:33:30.393396+0000","last_peered":"2024-04-26T20:33:30.393396+0000","last_clean":"2024-04-26T20:33:30.393396+0000","last_became_active":"2024-04-26T20:32:30.381648+0000","last_became_peered":"2024-04-26T20:32:30.381648+0000","last_unstale":"2024-04-26T20:33:30.393396+0000","last_undegraded":"2024-04-26T20:33:30.393396+0000","last_fullsized":"2024-04-26T20:33:30.393396+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:28.036984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.392219+0000","last_change":"2024-04-26T20:32:25.275597+0000","last_active":"2024-04-26T20:33:30.392219+0000","last_peered":"2024-04-26T20:33:30.392219+0000","last_clean":"2024-04-26T20:33:30.392219+0000","last_became_active":"2024-04-26T20:32:25.275256+0000","last_became_peered":"2024-04-26T20:32:25.275256+0000","last_unstale":"2024-04-26T20:33:30.392219+0000","last_undegraded":"2024-04-26T20:33:30.392219+0000","last_fullsized":"2024-04-26T20:33:30.392219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:38:20.354879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395715+0000","last_change":"2024-04-26T20:32:26.236885+0000","last_active":"2024-04-26T20:33:30.395715+0000","last_peered":"2024-04-26T20:33:30.395715+0000","last_clean":"2024-04-26T20:33:30.395715+0000","last_became_active":"2024-04-26T20:32:26.236581+0000","last_became_peered":"2024-04-26T20:32:26.236581+0000","last_unstale":"2024-04-26T20:33:30.395715+0000","last_undegraded":"2024-04-26T20:33:30.395715+0000","last_fullsized":"2024-04-26T20:33:30.395715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:17:35.082564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391292+0000","last_change":"2024-04-26T20:32:30.380156+0000","last_active":"2024-04-26T20:33:30.391292+0000","last_peered":"2024-04-26T20:33:30.391292+0000","last_clean":"2024-04-26T20:33:30.391292+0000","last_became_active":"2024-04-26T20:32:30.379240+0000","last_became_peered":"2024-04-26T20:32:30.379240+0000","last_unstale":"2024-04-26T20:33:30.391292+0000","last_undegraded":"2024-04-26T20:33:30.391292+0000","last_fullsized":"2024-04-26T20:33:30.391292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:29.324545+0000","last_clean_scrub_stamp":"2024-04-26T20:32:29.324545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:42:39.650923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.400793+0000","last_change":"2024-04-26T20:32:25.298850+0000","last_active":"2024-04-26T20:33:30.400793+0000","last_peered":"2024-04-26T20:33:30.400793+0000","last_clean":"2024-04-26T20:33:30.400793+0000","last_became_active":"2024-04-26T20:32:25.298171+0000","last_became_peered":"2024-04-26T20:32:25.298171+0000","last_unstale":"2024-04-26T20:33:30.400793+0000","last_undegraded":"2024-04-26T20:33:30.400793+0000","last_fullsized":"2024-04-26T20:33:30.400793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:24.232513+0000","last_clean_scrub_stamp":"2024-04-26T20:32:24.232513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:17:03.723432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.395843+0000","last_change":"2024-04-26T20:32:26.222873+0000","last_active":"2024-04-26T20:33:30.395843+0000","last_peered":"2024-04-26T20:33:30.395843+0000","last_clean":"2024-04-26T20:33:30.395843+0000","last_became_active":"2024-04-26T20:32:26.222487+0000","last_became_peered":"2024-04-26T20:32:26.222487+0000","last_unstale":"2024-04-26T20:33:30.395843+0000","last_undegraded":"2024-04-26T20:33:30.395843+0000","last_fullsized":"2024-04-26T20:33:30.395843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:17.521537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.384742+0000","last_change":"2024-04-26T20:32:26.227092+0000","last_active":"2024-04-26T20:33:30.384742+0000","last_peered":"2024-04-26T20:33:30.384742+0000","last_clean":"2024-04-26T20:33:30.384742+0000","last_became_active":"2024-04-26T20:32:26.226748+0000","last_became_peered":"2024-04-26T20:32:26.226748+0000","last_unstale":"2024-04-26T20:33:30.384742+0000","last_undegraded":"2024-04-26T20:33:30.384742+0000","last_fullsized":"2024-04-26T20:33:30.384742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:25.141131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391998+0000","last_change":"2024-04-26T20:32:26.220388+0000","last_active":"2024-04-26T20:33:30.391998+0000","last_peered":"2024-04-26T20:33:30.391998+0000","last_clean":"2024-04-26T20:33:30.391998+0000","last_became_active":"2024-04-26T20:32:26.219999+0000","last_became_peered":"2024-04-26T20:32:26.219999+0000","last_unstale":"2024-04-26T20:33:30.391998+0000","last_undegraded":"2024-04-26T20:33:30.391998+0000","last_fullsized":"2024-04-26T20:33:30.391998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:11.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.392780+0000","last_change":"2024-04-26T20:32:26.240122+0000","last_active":"2024-04-26T20:33:30.392780+0000","last_peered":"2024-04-26T20:33:30.392780+0000","last_clean":"2024-04-26T20:33:30.392780+0000","last_became_active":"2024-04-26T20:32:26.239746+0000","last_became_peered":"2024-04-26T20:32:26.239746+0000","last_unstale":"2024-04-26T20:33:30.392780+0000","last_undegraded":"2024-04-26T20:33:30.392780+0000","last_fullsized":"2024-04-26T20:33:30.392780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:10.243469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.394225+0000","last_change":"2024-04-26T20:32:26.254171+0000","last_active":"2024-04-26T20:33:30.394225+0000","last_peered":"2024-04-26T20:33:30.394225+0000","last_clean":"2024-04-26T20:33:30.394225+0000","last_became_active":"2024-04-26T20:32:26.253775+0000","last_became_peered":"2024-04-26T20:32:26.253775+0000","last_unstale":"2024-04-26T20:33:30.394225+0000","last_undegraded":"2024-04-26T20:33:30.394225+0000","last_fullsized":"2024-04-26T20:33:30.394225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:19:57.968344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.399014+0000","last_change":"2024-04-26T20:32:26.256475+0000","last_active":"2024-04-26T20:33:30.399014+0000","last_peered":"2024-04-26T20:33:30.399014+0000","last_clean":"2024-04-26T20:33:30.399014+0000","last_became_active":"2024-04-26T20:32:26.256119+0000","last_became_peered":"2024-04-26T20:32:26.256119+0000","last_unstale":"2024-04-26T20:33:30.399014+0000","last_undegraded":"2024-04-26T20:33:30.399014+0000","last_fullsized":"2024-04-26T20:33:30.399014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:39:31.456077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385136+0000","last_change":"2024-04-26T20:32:26.229251+0000","last_active":"2024-04-26T20:33:30.385136+0000","last_peered":"2024-04-26T20:33:30.385136+0000","last_clean":"2024-04-26T20:33:30.385136+0000","last_became_active":"2024-04-26T20:32:26.228847+0000","last_became_peered":"2024-04-26T20:32:26.228847+0000","last_unstale":"2024-04-26T20:33:30.385136+0000","last_undegraded":"2024-04-26T20:33:30.385136+0000","last_fullsized":"2024-04-26T20:33:30.385136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:07:28.489681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.392696+0000","last_change":"2024-04-26T20:32:26.223939+0000","last_active":"2024-04-26T20:33:30.392696+0000","last_peered":"2024-04-26T20:33:30.392696+0000","last_clean":"2024-04-26T20:33:30.392696+0000","last_became_active":"2024-04-26T20:32:26.223602+0000","last_became_peered":"2024-04-26T20:32:26.223602+0000","last_unstale":"2024-04-26T20:33:30.392696+0000","last_undegraded":"2024-04-26T20:33:30.392696+0000","last_fullsized":"2024-04-26T20:33:30.392696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:10:54.588411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.389339+0000","last_change":"2024-04-26T20:32:26.229912+0000","last_active":"2024-04-26T20:33:30.389339+0000","last_peered":"2024-04-26T20:33:30.389339+0000","last_clean":"2024-04-26T20:33:30.389339+0000","last_became_active":"2024-04-26T20:32:26.228498+0000","last_became_peered":"2024-04-26T20:32:26.228498+0000","last_unstale":"2024-04-26T20:33:30.389339+0000","last_undegraded":"2024-04-26T20:33:30.389339+0000","last_fullsized":"2024-04-26T20:33:30.389339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:17:53.302225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.391674+0000","last_change":"2024-04-26T20:32:26.232087+0000","last_active":"2024-04-26T20:33:30.391674+0000","last_peered":"2024-04-26T20:33:30.391674+0000","last_clean":"2024-04-26T20:33:30.391674+0000","last_became_active":"2024-04-26T20:32:26.231525+0000","last_became_peered":"2024-04-26T20:32:26.231525+0000","last_unstale":"2024-04-26T20:33:30.391674+0000","last_undegraded":"2024-04-26T20:33:30.391674+0000","last_fullsized":"2024-04-26T20:33:30.391674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:05.326866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.385625+0000","last_change":"2024-04-26T20:32:26.224341+0000","last_active":"2024-04-26T20:33:30.385625+0000","last_peered":"2024-04-26T20:33:30.385625+0000","last_clean":"2024-04-26T20:33:30.385625+0000","last_became_active":"2024-04-26T20:32:26.223888+0000","last_became_peered":"2024-04-26T20:32:26.223888+0000","last_unstale":"2024-04-26T20:33:30.385625+0000","last_undegraded":"2024-04-26T20:33:30.385625+0000","last_fullsized":"2024-04-26T20:33:30.385625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:21.103180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.410671+0000","last_change":"2024-04-26T20:32:26.231255+0000","last_active":"2024-04-26T20:33:30.410671+0000","last_peered":"2024-04-26T20:33:30.410671+0000","last_clean":"2024-04-26T20:33:30.410671+0000","last_became_active":"2024-04-26T20:32:26.230823+0000","last_became_peered":"2024-04-26T20:32:26.230823+0000","last_unstale":"2024-04-26T20:33:30.410671+0000","last_undegraded":"2024-04-26T20:33:30.410671+0000","last_fullsized":"2024-04-26T20:33:30.410671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:34.066151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.397785+0000","last_change":"2024-04-26T20:32:26.246452+0000","last_active":"2024-04-26T20:33:30.397785+0000","last_peered":"2024-04-26T20:33:30.397785+0000","last_clean":"2024-04-26T20:33:30.397785+0000","last_became_active":"2024-04-26T20:32:26.246109+0000","last_became_peered":"2024-04-26T20:32:26.246109+0000","last_unstale":"2024-04-26T20:33:30.397785+0000","last_undegraded":"2024-04-26T20:33:30.397785+0000","last_fullsized":"2024-04-26T20:33:30.397785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:42:21.617776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T20:33:30.405491+0000","last_change":"2024-04-26T20:33:30.405491+0000","last_active":"2024-04-26T20:33:30.405491+0000","last_peered":"2024-04-26T20:33:30.405491+0000","last_clean":"2024-04-26T20:33:30.405491+0000","last_became_active":"2024-04-26T20:33:30.405010+0000","last_became_peered":"2024-04-26T20:33:30.405010+0000","last_unstale":"2024-04-26T20:33:30.405491+0000","last_undegraded":"2024-04-26T20:33:30.405491+0000","last_fullsized":"2024-04-26T20:33:30.405491+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T20:32:25.235399+0000","last_clean_scrub_stamp":"2024-04-26T20:32:25.235399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:23:13.675737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61248,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":867,"num_read_kb":499,"num_write":244,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":501449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28704,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716448,"statfs":{"total":95995035648,"available":95965642752,"internally_reserved":0,"allocated":1720320,"data_stored":1143091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":505475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070519,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":501757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1650688,"data_stored":1097863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":496883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":552739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":498339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":552337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":496591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":501579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1658880,"data_stored":1093445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":56578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":56176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":56562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T20:34:24.598 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T20:34:24.598 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T20:34:24.598 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T20:34:24.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T20:34:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:24 smithi112 ceph-mon[32769]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:34:25.080 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:25.111 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:24 smithi102 ceph-mon[28094]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:34:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:24 smithi130 ceph-mon[32704]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T20:34:26.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:25 smithi112 ceph-mon[32769]: pgmap v379: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:34:26.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:25 smithi130 ceph-mon[32704]: pgmap v379: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:34:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:25 smithi102 ceph-mon[28094]: pgmap v379: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:34:27.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.358+0000 7f7f42e00700 1 -- 172.21.15.102:0/1338447942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f3c109a50 msgr2=0x7f7f3c109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:27.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.358+0000 7f7f42e00700 1 --2- 172.21.15.102:0/1338447942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f3c109a50 0x7f7f3c109ed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009990 tx=0x7f7f28009ca0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.358+0000 7f7f42e00700 1 -- 172.21.15.102:0/1338447942 shutdown_connections 2024-04-26T20:34:27.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.358+0000 7f7f42e00700 1 --2- 172.21.15.102:0/1338447942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f3c10a410 0x7f7f3c116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.358+0000 7f7f42e00700 1 --2- 172.21.15.102:0/1338447942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f3c10fa50 0x7f7f3c10fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.358+0000 7f7f42e00700 1 --2- 172.21.15.102:0/1338447942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f3c109a50 0x7f7f3c109ed0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.358+0000 7f7f42e00700 1 -- 172.21.15.102:0/1338447942 >> 172.21.15.102:0/1338447942 conn(0x7f7f3c076bf0 msgr2=0x7f7f3c108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:27.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.358+0000 7f7f42e00700 1 -- 172.21.15.102:0/1338447942 shutdown_connections 2024-04-26T20:34:27.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.358+0000 7f7f42e00700 1 -- 172.21.15.102:0/1338447942 wait complete. 2024-04-26T20:34:27.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.360+0000 7f7f42e00700 1 Processor -- start 2024-04-26T20:34:27.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.360+0000 7f7f42e00700 1 -- start start 2024-04-26T20:34:27.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.360+0000 7f7f42e00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f3c109a50 0x7f7f3c1a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:27.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.360+0000 7f7f42e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f3c10a410 0x7f7f3c1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:27.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.360+0000 7f7f42e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f3c10fa50 0x7f7f3c1a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:27.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.360+0000 7f7f42e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c119c30 con 0x7f7f3c10a410 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.360+0000 7f7f42e00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c119ab0 con 0x7f7f3c109a50 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.360+0000 7f7f42e00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c119db0 con 0x7f7f3c10fa50 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f40e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f3c10fa50 0x7f7f3c1a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f3be00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f3c109a50 0x7f7f3c1a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f40e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f3c10fa50 0x7f7f3c1a49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55486/0 (socket says 172.21.15.102:55486) 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f40e00700 1 -- 172.21.15.102:0/3036131345 learned_addr learned my addr 172.21.15.102:0/3036131345 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f3b400700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f3c10a410 0x7f7f3c1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f40e00700 1 -- 172.21.15.102:0/3036131345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f3c109a50 msgr2=0x7f7f3c1a00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f40e00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f3c109a50 0x7f7f3c1a00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f40e00700 1 -- 172.21.15.102:0/3036131345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f3c10a410 msgr2=0x7f7f3c1a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f40e00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f3c10a410 0x7f7f3c1a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f40e00700 1 -- 172.21.15.102:0/3036131345 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3c1a50f0 con 0x7f7f3c10fa50 2024-04-26T20:34:27.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.361+0000 7f7f40e00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f3c10fa50 0x7f7f3c1a49b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400ef30 tx=0x7f7f3400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:27.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.362+0000 7f7f38c00700 1 -- 172.21.15.102:0/3036131345 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f340092e0 con 0x7f7f3c10fa50 2024-04-26T20:34:27.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.362+0000 7f7f38c00700 1 -- 172.21.15.102:0/3036131345 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f34004bb0 con 0x7f7f3c10fa50 2024-04-26T20:34:27.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.362+0000 7f7f42e00700 1 -- 172.21.15.102:0/3036131345 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f3c1a5380 con 0x7f7f3c10fa50 2024-04-26T20:34:27.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.362+0000 7f7f38c00700 1 -- 172.21.15.102:0/3036131345 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34004020 con 0x7f7f3c10fa50 2024-04-26T20:34:27.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.362+0000 7f7f42e00700 1 -- 172.21.15.102:0/3036131345 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f3c1acc60 con 0x7f7f3c10fa50 2024-04-26T20:34:27.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.363+0000 7f7f42e00700 1 -- 172.21.15.102:0/3036131345 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f3c06a110 con 0x7f7f3c10fa50 2024-04-26T20:34:27.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.365+0000 7f7f38c00700 1 -- 172.21.15.102:0/3036131345 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f7f3401b030 con 0x7f7f3c10fa50 2024-04-26T20:34:27.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.366+0000 7f7f38c00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7f200a3d50 0x7f7f200a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:27.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.366+0000 7f7f38c00700 1 -- 172.21.15.102:0/3036131345 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7f340d0360 con 0x7f7f3c10fa50 2024-04-26T20:34:27.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.367+0000 7f7f3be00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7f200a3d50 0x7f7f200a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:27.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.367+0000 7f7f3be00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7f200a3d50 0x7f7f200a6210 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009be0 tx=0x7f7f2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:27.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.371+0000 7f7f38c00700 1 -- 172.21.15.102:0/3036131345 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7f34022020 con 0x7f7f3c10fa50 2024-04-26T20:34:27.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.649+0000 7f7f42e00700 1 -- 172.21.15.102:0/3036131345 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f7f3c02e490 con 0x7f7f3c10fa50 2024-04-26T20:34:27.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.650+0000 7f7f38c00700 1 -- 172.21.15.102:0/3036131345 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f7f34010040 con 0x7f7f3c10fa50 2024-04-26T20:34:27.651 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:27.652 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T20:34:27.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 -- 172.21.15.102:0/3036131345 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7f200a3d50 msgr2=0x7f7f200a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:27.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7f200a3d50 0x7f7f200a6210 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009be0 tx=0x7f7f2c009260 comp rx=0 tx=0).stop 2024-04-26T20:34:27.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 -- 172.21.15.102:0/3036131345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f3c10fa50 msgr2=0x7f7f3c1a49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:27.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f3c10fa50 0x7f7f3c1a49b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400ef30 tx=0x7f7f3400c3a0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 -- 172.21.15.102:0/3036131345 shutdown_connections 2024-04-26T20:34:27.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7f200a3d50 0x7f7f200a6210 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f3c10fa50 0x7f7f3c1a49b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7f3c109a50 0x7f7f3c1a00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 --2- 172.21.15.102:0/3036131345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f3c10a410 0x7f7f3c1a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:27.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 -- 172.21.15.102:0/3036131345 >> 172.21.15.102:0/3036131345 conn(0x7f7f3c076bf0 msgr2=0x7f7f3c0771b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:27.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 -- 172.21.15.102:0/3036131345 shutdown_connections 2024-04-26T20:34:27.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:27.653+0000 7f7f42e00700 1 -- 172.21.15.102:0/3036131345 wait complete. 2024-04-26T20:34:28.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:27 smithi112 ceph-mon[32769]: pgmap v380: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:34:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:27 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3036131345' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T20:34:28.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:27 smithi102 ceph-mon[28094]: pgmap v380: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:34:28.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:27 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3036131345' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T20:34:28.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:27 smithi130 ceph-mon[32704]: pgmap v380: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:34:28.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:27 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3036131345' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T20:34:28.423 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T20:34:28.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:34:28.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.632+0000 7ff59ac00640 1 -- 172.21.15.102:0/1709359485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5941064f0 msgr2=0x7ff594106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:28.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.632+0000 7ff59ac00640 1 --2- 172.21.15.102:0/1709359485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5941064f0 0x7ff594106970 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009ca0 tx=0x7ff58c02a8c0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.632+0000 7ff59ac00640 1 -- 172.21.15.102:0/1709359485 shutdown_connections 2024-04-26T20:34:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.632+0000 7ff59ac00640 1 --2- 172.21.15.102:0/1709359485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5941064f0 0x7ff594106970 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.632+0000 7ff59ac00640 1 --2- 172.21.15.102:0/1709359485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff594103b40 0x7ff594105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.632+0000 7ff59ac00640 1 --2- 172.21.15.102:0/1709359485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5941011d0 0x7ff594103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.632+0000 7ff59ac00640 1 -- 172.21.15.102:0/1709359485 >> 172.21.15.102:0/1709359485 conn(0x7ff5940fcdd0 msgr2=0x7ff5940ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.632+0000 7ff59ac00640 1 -- 172.21.15.102:0/1709359485 shutdown_connections 2024-04-26T20:34:28.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.632+0000 7ff59ac00640 1 -- 172.21.15.102:0/1709359485 wait complete. 2024-04-26T20:34:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.634+0000 7ff59ac00640 1 Processor -- start 2024-04-26T20:34:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.634+0000 7ff59ac00640 1 -- start start 2024-04-26T20:34:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.634+0000 7ff59ac00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5941011d0 0x7ff59419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:28.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.634+0000 7ff59ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff594103b40 0x7ff59419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.634+0000 7ff59ac00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5941064f0 0x7ff5941a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.634+0000 7ff59ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff594114b00 con 0x7ff594103b40 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.634+0000 7ff59ac00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff594114980 con 0x7ff5941064f0 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.634+0000 7ff59ac00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff594114c80 con 0x7ff5941011d0 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff593e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5941011d0 0x7ff59419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff593e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5941011d0 0x7ff59419f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55502/0 (socket says 172.21.15.102:55502) 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff593e00640 1 -- 172.21.15.102:0/1786892286 learned_addr learned my addr 172.21.15.102:0/1786892286 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff593400640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff594103b40 0x7ff59419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff598c00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5941064f0 0x7ff5941a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff593e00640 1 -- 172.21.15.102:0/1786892286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5941064f0 msgr2=0x7ff5941a66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff593e00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5941064f0 0x7ff5941a66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff593e00640 1 -- 172.21.15.102:0/1786892286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff594103b40 msgr2=0x7ff59419f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff593e00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff594103b40 0x7ff59419f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff593e00640 1 -- 172.21.15.102:0/1786892286 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5941a6d10 con 0x7ff5941011d0 2024-04-26T20:34:28.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.635+0000 7ff593e00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5941011d0 0x7ff59419f0f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff58400e9f0 tx=0x7ff58400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:28.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.636+0000 7ff590c00640 1 -- 172.21.15.102:0/1786892286 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58400cd80 con 0x7ff5941011d0 2024-04-26T20:34:28.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.636+0000 7ff590c00640 1 -- 172.21.15.102:0/1786892286 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5840108e0 con 0x7ff5941011d0 2024-04-26T20:34:28.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.636+0000 7ff590c00640 1 -- 172.21.15.102:0/1786892286 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584018980 con 0x7ff5941011d0 2024-04-26T20:34:28.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.636+0000 7ff59ac00640 1 -- 172.21.15.102:0/1786892286 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5941a6fa0 con 0x7ff5941011d0 2024-04-26T20:34:28.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.636+0000 7ff59ac00640 1 -- 172.21.15.102:0/1786892286 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5941a7410 con 0x7ff5941011d0 2024-04-26T20:34:28.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.637+0000 7ff59ac00640 1 -- 172.21.15.102:0/1786892286 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5940fe8c0 con 0x7ff5941011d0 2024-04-26T20:34:28.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.639+0000 7ff590c00640 1 -- 172.21.15.102:0/1786892286 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff584010430 con 0x7ff5941011d0 2024-04-26T20:34:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.640+0000 7ff590c00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff5780a3cc0 0x7ff5780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.640+0000 7ff590c00640 1 -- 172.21.15.102:0/1786892286 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff5840d0bb0 con 0x7ff5941011d0 2024-04-26T20:34:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.641+0000 7ff593400640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff5780a3cc0 0x7ff5780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.641+0000 7ff593400640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff5780a3cc0 0x7ff5780a6180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff5941a0600 tx=0x7ff580005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:28.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.645+0000 7ff590c00640 1 -- 172.21.15.102:0/1786892286 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff58409ca20 con 0x7ff5941011d0 2024-04-26T20:34:28.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.846+0000 7ff59ac00640 1 -- 172.21.15.102:0/1786892286 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff59402dd80 con 0x7ff5941011d0 2024-04-26T20:34:28.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.848+0000 7ff590c00640 1 -- 172.21.15.102:0/1786892286 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24636 (secure 0 0 0) 0x7ff5840a0d20 con 0x7ff5941011d0 2024-04-26T20:34:28.850 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:28.850 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":91,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:33:30.376705+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:34:28.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.850+0000 7ff59ac00640 1 -- 172.21.15.102:0/1786892286 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff5780a3cc0 msgr2=0x7ff5780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:28.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.850+0000 7ff59ac00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff5780a3cc0 0x7ff5780a6180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff5941a0600 tx=0x7ff580005f90 comp rx=0 tx=0).stop 2024-04-26T20:34:28.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.850+0000 7ff59ac00640 1 -- 172.21.15.102:0/1786892286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5941011d0 msgr2=0x7ff59419f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:28.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.850+0000 7ff59ac00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5941011d0 0x7ff59419f0f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff58400e9f0 tx=0x7ff58400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.851+0000 7ff59ac00640 1 -- 172.21.15.102:0/1786892286 shutdown_connections 2024-04-26T20:34:28.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.851+0000 7ff59ac00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff5780a3cc0 0x7ff5780a6180 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.851+0000 7ff59ac00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff594103b40 0x7ff59419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.851+0000 7ff59ac00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5941064f0 0x7ff5941a66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.851+0000 7ff59ac00640 1 --2- 172.21.15.102:0/1786892286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5941011d0 0x7ff59419f0f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:28.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.851+0000 7ff59ac00640 1 -- 172.21.15.102:0/1786892286 >> 172.21.15.102:0/1786892286 conn(0x7ff5940fcdd0 msgr2=0x7ff5941044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:28.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.851+0000 7ff59ac00640 1 -- 172.21.15.102:0/1786892286 shutdown_connections 2024-04-26T20:34:28.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:28.851+0000 7ff59ac00640 1 -- 172.21.15.102:0/1786892286 wait complete. 2024-04-26T20:34:28.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:27:10.527933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:32:23.439237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:32:24.702263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:32:28.348785+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:34:28.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:34:29.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.078+0000 7f4317200640 1 -- 172.21.15.102:0/2867428902 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4318100f20 msgr2=0x7f4318107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.078+0000 7f4317200640 1 --2- 172.21.15.102:0/2867428902 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4318100f20 0x7f4318107f20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4304009900 tx=0x7f430402a400 comp rx=0 tx=0).stop 2024-04-26T20:34:29.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.078+0000 7f4317200640 1 -- 172.21.15.102:0/2867428902 shutdown_connections 2024-04-26T20:34:29.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.078+0000 7f4317200640 1 --2- 172.21.15.102:0/2867428902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43181084b0 0x7f431810a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.078+0000 7f4317200640 1 --2- 172.21.15.102:0/2867428902 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4318100f20 0x7f4318107f20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.078+0000 7f4317200640 1 --2- 172.21.15.102:0/2867428902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43181005c0 0x7f43181009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.078+0000 7f4317200640 1 -- 172.21.15.102:0/2867428902 >> 172.21.15.102:0/2867428902 conn(0x7f43180fbff0 msgr2=0x7f43180fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:29.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.078+0000 7f4317200640 1 -- 172.21.15.102:0/2867428902 shutdown_connections 2024-04-26T20:34:29.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.078+0000 7f4317200640 1 -- 172.21.15.102:0/2867428902 wait complete. 2024-04-26T20:34:29.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.079+0000 7f4317200640 1 Processor -- start 2024-04-26T20:34:29.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.079+0000 7f4317200640 1 -- start start 2024-04-26T20:34:29.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4317200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43181005c0 0x7f4318103820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4317200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4318100f20 0x7f4318101eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4317200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43181084b0 0x7f43181023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4317200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f431810e390 con 0x7f4318100f20 2024-04-26T20:34:29.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4317200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f431810e210 con 0x7f43181005c0 2024-04-26T20:34:29.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4317200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f431810e510 con 0x7f43181084b0 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4315e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43181005c0 0x7f4318103820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4315400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4318100f20 0x7f4318101eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4315e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43181005c0 0x7f4318103820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:40410/0 (socket says 172.21.15.102:40410) 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4315e00640 1 -- 172.21.15.102:0/1837923272 learned_addr learned my addr 172.21.15.102:0/1837923272 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4316800640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43181084b0 0x7f43181023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4315e00640 1 -- 172.21.15.102:0/1837923272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43181084b0 msgr2=0x7f43181023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4315e00640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43181084b0 0x7f43181023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4315e00640 1 -- 172.21.15.102:0/1837923272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4318100f20 msgr2=0x7f4318101eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.080+0000 7f4315e00640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4318100f20 0x7f4318101eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.081+0000 7f4315e00640 1 -- 172.21.15.102:0/1837923272 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4318102cb0 con 0x7f43181005c0 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.081+0000 7f4315e00640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43181005c0 0x7f4318103820 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f430800d950 tx=0x7f430800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.081+0000 7f430ea00640 1 -- 172.21.15.102:0/1837923272 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308017030 con 0x7f43181005c0 2024-04-26T20:34:29.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.081+0000 7f4317200640 1 -- 172.21.15.102:0/1837923272 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43181a2c70 con 0x7f43181005c0 2024-04-26T20:34:29.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.081+0000 7f430ea00640 1 -- 172.21.15.102:0/1837923272 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4308004020 con 0x7f43181005c0 2024-04-26T20:34:29.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.081+0000 7f430ea00640 1 -- 172.21.15.102:0/1837923272 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43080047a0 con 0x7f43181005c0 2024-04-26T20:34:29.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.081+0000 7f4317200640 1 -- 172.21.15.102:0/1837923272 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43181a3190 con 0x7f43181005c0 2024-04-26T20:34:29.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.082+0000 7f4317200640 1 -- 172.21.15.102:0/1837923272 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4318106d30 con 0x7f43181005c0 2024-04-26T20:34:29.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.085+0000 7f430ea00640 1 -- 172.21.15.102:0/1837923272 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f4308009a30 con 0x7f43181005c0 2024-04-26T20:34:29.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.086+0000 7f430ea00640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42fc0a3c40 0x7f42fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.087+0000 7f430ea00640 1 -- 172.21.15.102:0/1837923272 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f43080d0550 con 0x7f43181005c0 2024-04-26T20:34:29.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.087+0000 7f4315400640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42fc0a3c40 0x7f42fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.087+0000 7f4315400640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42fc0a3c40 0x7f42fc0a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4304005ab0 tx=0x7f4304005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:29.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.088+0000 7f430ea00640 1 -- 172.21.15.102:0/1837923272 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f430809ca10 con 0x7f43181005c0 2024-04-26T20:34:29.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.290+0000 7f4317200640 1 -- 172.21.15.102:0/1837923272 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f431802dd60 con 0x7f43181005c0 2024-04-26T20:34:29.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.291+0000 7f430ea00640 1 -- 172.21.15.102:0/1837923272 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f430809c310 con 0x7f43181005c0 2024-04-26T20:34:29.292 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-26T20:34:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 -- 172.21.15.102:0/1837923272 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42fc0a3c40 msgr2=0x7f42fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42fc0a3c40 0x7f42fc0a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4304005ab0 tx=0x7f4304005a20 comp rx=0 tx=0).stop 2024-04-26T20:34:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 -- 172.21.15.102:0/1837923272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43181005c0 msgr2=0x7f4318103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43181005c0 0x7f4318103820 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f430800d950 tx=0x7f430800de10 comp rx=0 tx=0).stop 2024-04-26T20:34:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 -- 172.21.15.102:0/1837923272 shutdown_connections 2024-04-26T20:34:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42fc0a3c40 0x7f42fc0a6100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4318100f20 0x7f4318101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43181084b0 0x7f43181023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 --2- 172.21.15.102:0/1837923272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43181005c0 0x7f4318103820 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 -- 172.21.15.102:0/1837923272 >> 172.21.15.102:0/1837923272 conn(0x7f43180fbff0 msgr2=0x7f4318103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 -- 172.21.15.102:0/1837923272 shutdown_connections 2024-04-26T20:34:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.294+0000 7f4317200640 1 -- 172.21.15.102:0/1837923272 wait complete. 2024-04-26T20:34:29.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:34:29.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.521+0000 7fd1b9800640 1 -- 172.21.15.102:0/2814338825 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1b4106620 msgr2=0x7fd1b410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.521+0000 7fd1b9800640 1 --2- 172.21.15.102:0/2814338825 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1b4106620 0x7fd1b410ceb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009ca0 tx=0x7fd1ac02a8c0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.521+0000 7fd1b9800640 1 -- 172.21.15.102:0/2814338825 shutdown_connections 2024-04-26T20:34:29.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.521+0000 7fd1b9800640 1 --2- 172.21.15.102:0/2814338825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4105c60 0x7fd1b41060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.521+0000 7fd1b9800640 1 --2- 172.21.15.102:0/2814338825 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1b4106620 0x7fd1b410ceb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.521+0000 7fd1b9800640 1 --2- 172.21.15.102:0/2814338825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b4104a70 0x7fd1b4104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.521+0000 7fd1b9800640 1 -- 172.21.15.102:0/2814338825 >> 172.21.15.102:0/2814338825 conn(0x7fd1b40fffe0 msgr2=0x7fd1b4102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:29.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.521+0000 7fd1b9800640 1 -- 172.21.15.102:0/2814338825 shutdown_connections 2024-04-26T20:34:29.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.521+0000 7fd1b9800640 1 -- 172.21.15.102:0/2814338825 wait complete. 2024-04-26T20:34:29.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.522+0000 7fd1b9800640 1 Processor -- start 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.522+0000 7fd1b9800640 1 -- start start 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.522+0000 7fd1b9800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b4104a70 0x7fd1b419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.522+0000 7fd1b9800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1b4105c60 0x7fd1b419b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.522+0000 7fd1b9800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4106620 0x7fd1b41a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.522+0000 7fd1b9800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b41105f0 con 0x7fd1b4106620 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.522+0000 7fd1b9800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b4110470 con 0x7fd1b4104a70 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.522+0000 7fd1b9800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b4110770 con 0x7fd1b4105c60 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.522+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b4104a70 0x7fd1b419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.523+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b4104a70 0x7fd1b419aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:40434/0 (socket says 172.21.15.102:40434) 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.523+0000 7fd1b3e00640 1 -- 172.21.15.102:0/2879515681 learned_addr learned my addr 172.21.15.102:0/2879515681 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.523+0000 7fd1b8e00640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4106620 0x7fd1b41a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.523+0000 7fd1ab400640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1b4105c60 0x7fd1b419b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.523+0000 7fd1ab400640 1 -- 172.21.15.102:0/2879515681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b4104a70 msgr2=0x7fd1b419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.523+0000 7fd1ab400640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b4104a70 0x7fd1b419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.523+0000 7fd1ab400640 1 -- 172.21.15.102:0/2879515681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4106620 msgr2=0x7fd1b41a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.523+0000 7fd1ab400640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4106620 0x7fd1b41a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.523+0000 7fd1ab400640 1 -- 172.21.15.102:0/2879515681 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b41a28e0 con 0x7fd1b4105c60 2024-04-26T20:34:29.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.523+0000 7fd1ab400640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1b4105c60 0x7fd1b419b1e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00e9f0 tx=0x7fd19c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:29.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.524+0000 7fd1b1600640 1 -- 172.21.15.102:0/2879515681 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00ccc0 con 0x7fd1b4105c60 2024-04-26T20:34:29.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.524+0000 7fd1b9800640 1 -- 172.21.15.102:0/2879515681 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b41a2bd0 con 0x7fd1b4105c60 2024-04-26T20:34:29.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.524+0000 7fd1b1600640 1 -- 172.21.15.102:0/2879515681 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd19c0108e0 con 0x7fd1b4105c60 2024-04-26T20:34:29.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.524+0000 7fd1b1600640 1 -- 172.21.15.102:0/2879515681 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c018880 con 0x7fd1b4105c60 2024-04-26T20:34:29.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.524+0000 7fd1b9800640 1 -- 172.21.15.102:0/2879515681 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b41a3120 con 0x7fd1b4105c60 2024-04-26T20:34:29.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.525+0000 7fd1b9800640 1 -- 172.21.15.102:0/2879515681 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b410bd30 con 0x7fd1b4105c60 2024-04-26T20:34:29.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.527+0000 7fd1b1600640 1 -- 172.21.15.102:0/2879515681 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fd19c010430 con 0x7fd1b4105c60 2024-04-26T20:34:29.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.528+0000 7fd1b1600640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd1980a3c40 0x7fd1980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.528+0000 7fd1b1600640 1 -- 172.21.15.102:0/2879515681 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd19c0d1110 con 0x7fd1b4105c60 2024-04-26T20:34:29.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.529+0000 7fd1b3e00640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd1980a3c40 0x7fd1980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.529+0000 7fd1b3e00640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd1980a3c40 0x7fd1980a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd1b419bb70 tx=0x7fd1a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:29.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.531+0000 7fd1b1600640 1 -- 172.21.15.102:0/2879515681 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd19c09cf80 con 0x7fd1b4105c60 2024-04-26T20:34:29.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.740+0000 7fd1b9800640 1 -- 172.21.15.102:0/2879515681 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd1b402dad0 con 0x7fd1b4105c60 2024-04-26T20:34:29.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.740+0000 7fd1b1600640 1 -- 172.21.15.102:0/2879515681 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7fd19c0a1280 con 0x7fd1b4105c60 2024-04-26T20:34:29.741 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:34:29.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 -- 172.21.15.102:0/2879515681 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd1980a3c40 msgr2=0x7fd1980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd1980a3c40 0x7fd1980a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd1b419bb70 tx=0x7fd1a4005f90 comp rx=0 tx=0).stop 2024-04-26T20:34:29.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 -- 172.21.15.102:0/2879515681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1b4105c60 msgr2=0x7fd1b419b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1b4105c60 0x7fd1b419b1e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00e9f0 tx=0x7fd19c00eeb0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 -- 172.21.15.102:0/2879515681 shutdown_connections 2024-04-26T20:34:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd1980a3c40 0x7fd1980a6100 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4106620 0x7fd1b41a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1b4105c60 0x7fd1b419b1e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 --2- 172.21.15.102:0/2879515681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1b4104a70 0x7fd1b419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 -- 172.21.15.102:0/2879515681 >> 172.21.15.102:0/2879515681 conn(0x7fd1b40fffe0 msgr2=0x7fd1b4108f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 -- 172.21.15.102:0/2879515681 shutdown_connections 2024-04-26T20:34:29.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.743+0000 7fd1b9800640 1 -- 172.21.15.102:0/2879515681 wait complete. 2024-04-26T20:34:29.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:34:29.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.963+0000 7f1d26a00640 1 -- 172.21.15.102:0/1305458272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d20104320 msgr2=0x7f1d2010abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.963+0000 7f1d26a00640 1 --2- 172.21.15.102:0/1305458272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d20104320 0x7f1d2010abb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009ca0 tx=0x7f1d1802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.963+0000 7f1d26a00640 1 -- 172.21.15.102:0/1305458272 shutdown_connections 2024-04-26T20:34:29.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.963+0000 7f1d26a00640 1 --2- 172.21.15.102:0/1305458272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d20104320 0x7f1d2010abb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.963+0000 7f1d26a00640 1 --2- 172.21.15.102:0/1305458272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d20103960 0x7f1d20103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.963+0000 7f1d26a00640 1 --2- 172.21.15.102:0/1305458272 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d20102770 0x7f1d20102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.963+0000 7f1d26a00640 1 -- 172.21.15.102:0/1305458272 >> 172.21.15.102:0/1305458272 conn(0x7f1d200fdce0 msgr2=0x7f1d20100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:29.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.963+0000 7f1d26a00640 1 -- 172.21.15.102:0/1305458272 shutdown_connections 2024-04-26T20:34:29.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.963+0000 7f1d26a00640 1 -- 172.21.15.102:0/1305458272 wait complete. 2024-04-26T20:34:29.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.964+0000 7f1d26a00640 1 Processor -- start 2024-04-26T20:34:29.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.964+0000 7f1d26a00640 1 -- start start 2024-04-26T20:34:29.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d26a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d20102770 0x7f1d2019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d26a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d20103960 0x7f1d2019b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d26a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d20104320 0x7f1d201a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d26a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2010e2d0 con 0x7f1d20102770 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d26a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2010e150 con 0x7f1d20104320 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d26a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2010e450 con 0x7f1d20103960 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d25600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d20102770 0x7f1d2019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d25600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d20102770 0x7f1d2019ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50074/0 (socket says 172.21.15.102:50074) 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d25600640 1 -- 172.21.15.102:0/1442170259 learned_addr learned my addr 172.21.15.102:0/1442170259 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d24c00640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d20103960 0x7f1d2019b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d26000640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d20104320 0x7f1d201a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d25600640 1 -- 172.21.15.102:0/1442170259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d20103960 msgr2=0x7f1d2019b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d25600640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d20103960 0x7f1d2019b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d25600640 1 -- 172.21.15.102:0/1442170259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d20104320 msgr2=0x7f1d201a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d25600640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d20104320 0x7f1d201a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d25600640 1 -- 172.21.15.102:0/1442170259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d201a2760 con 0x7f1d20102770 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.965+0000 7f1d24c00640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d20103960 0x7f1d2019b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:29.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.966+0000 7f1d25600640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d20102770 0x7f1d2019ab40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400e9c0 tx=0x7f1d1400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:29.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.966+0000 7f1d1e000640 1 -- 172.21.15.102:0/1442170259 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1400cd80 con 0x7f1d20102770 2024-04-26T20:34:29.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.966+0000 7f1d1e000640 1 -- 172.21.15.102:0/1442170259 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d140035f0 con 0x7f1d20102770 2024-04-26T20:34:29.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.966+0000 7f1d1e000640 1 -- 172.21.15.102:0/1442170259 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14010690 con 0x7f1d20102770 2024-04-26T20:34:29.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.966+0000 7f1d26a00640 1 -- 172.21.15.102:0/1442170259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d201a2a50 con 0x7f1d20102770 2024-04-26T20:34:29.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.966+0000 7f1d26a00640 1 -- 172.21.15.102:0/1442170259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d201a2fa0 con 0x7f1d20102770 2024-04-26T20:34:29.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.967+0000 7f1d26a00640 1 -- 172.21.15.102:0/1442170259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d20069670 con 0x7f1d20102770 2024-04-26T20:34:29.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.969+0000 7f1d1e000640 1 -- 172.21.15.102:0/1442170259 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f1d140037a0 con 0x7f1d20102770 2024-04-26T20:34:29.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.971+0000 7f1d1e000640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1d080a3c40 0x7f1d080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:29.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.971+0000 7f1d1e000640 1 -- 172.21.15.102:0/1442170259 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9179+0+0 (secure 0 0 0) 0x7f1d140d0d00 con 0x7f1d20102770 2024-04-26T20:34:29.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.971+0000 7f1d24c00640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1d080a3c40 0x7f1d080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:29.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.972+0000 7f1d24c00640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1d080a3c40 0x7f1d080a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1d2019c050 tx=0x7f1d0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:29.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:29.976+0000 7f1d1e000640 1 -- 172.21.15.102:0/1442170259 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1d1409d170 con 0x7f1d20102770 2024-04-26T20:34:30.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:29 smithi112 ceph-mon[32769]: pgmap v381: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:34:30.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:29 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1786892286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:34:30.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:29 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1837923272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:34:30.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-26T20:34:30.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-26T20:34:30.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-26T20:34:30.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:29 smithi130 ceph-mon[32704]: pgmap v381: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:34:30.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:29 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1786892286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:34:30.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:29 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1837923272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:34:30.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-26T20:34:30.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-26T20:34:30.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-26T20:34:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:29 smithi102 ceph-mon[28094]: pgmap v381: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:34:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:29 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1786892286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:34:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:29 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1837923272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:34:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-26T20:34:30.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-26T20:34:30.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-26T20:34:30.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.171+0000 7f1d26a00640 1 -- 172.21.15.102:0/1442170259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1d201a34e0 con 0x7f1d20102770 2024-04-26T20:34:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.172+0000 7f1d1e000640 1 -- 172.21.15.102:0/1442170259 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f1d1409ca70 con 0x7f1d20102770 2024-04-26T20:34:30.173 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-26T20:34:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 -- 172.21.15.102:0/1442170259 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1d080a3c40 msgr2=0x7f1d080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1d080a3c40 0x7f1d080a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1d2019c050 tx=0x7f1d0c005f90 comp rx=0 tx=0).stop 2024-04-26T20:34:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 -- 172.21.15.102:0/1442170259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d20102770 msgr2=0x7f1d2019ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d20102770 0x7f1d2019ab40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400e9c0 tx=0x7f1d1400ee80 comp rx=0 tx=0).stop 2024-04-26T20:34:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 -- 172.21.15.102:0/1442170259 shutdown_connections 2024-04-26T20:34:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1d080a3c40 0x7f1d080a6100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1d20104320 0x7f1d201a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d20103960 0x7f1d2019b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 --2- 172.21.15.102:0/1442170259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d20102770 0x7f1d2019ab40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 -- 172.21.15.102:0/1442170259 >> 172.21.15.102:0/1442170259 conn(0x7f1d200fdce0 msgr2=0x7f1d20074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 -- 172.21.15.102:0/1442170259 shutdown_connections 2024-04-26T20:34:30.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.175+0000 7f1d26a00640 1 -- 172.21.15.102:0/1442170259 wait complete. 2024-04-26T20:34:30.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:34:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.395+0000 7fd225400640 1 -- 172.21.15.102:0/860196648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd220101a70 msgr2=0x7fd220101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:30.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.395+0000 7fd225400640 1 --2- 172.21.15.102:0/860196648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd220101a70 0x7fd220101ed0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009860 tx=0x7fd20c02a340 comp rx=0 tx=0).stop 2024-04-26T20:34:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.395+0000 7fd225400640 1 -- 172.21.15.102:0/860196648 shutdown_connections 2024-04-26T20:34:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.395+0000 7fd225400640 1 --2- 172.21.15.102:0/860196648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd220102410 0x7fd220102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.395+0000 7fd225400640 1 --2- 172.21.15.102:0/860196648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd220101a70 0x7fd220101ed0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.395+0000 7fd225400640 1 --2- 172.21.15.102:0/860196648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd220101110 0x7fd220101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.395+0000 7fd225400640 1 -- 172.21.15.102:0/860196648 >> 172.21.15.102:0/860196648 conn(0x7fd2200fccd0 msgr2=0x7fd2200ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.395+0000 7fd225400640 1 -- 172.21.15.102:0/860196648 shutdown_connections 2024-04-26T20:34:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.395+0000 7fd225400640 1 -- 172.21.15.102:0/860196648 wait complete. 2024-04-26T20:34:30.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.396+0000 7fd225400640 1 Processor -- start 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.396+0000 7fd225400640 1 -- start start 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd225400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd220101110 0x7fd22019ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd225400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd220101a70 0x7fd22019b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd225400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd220102410 0x7fd2201a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd225400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd220110530 con 0x7fd220101a70 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd225400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2201103b0 con 0x7fd220102410 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd225400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd2201106b0 con 0x7fd220101110 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd21fe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd220101110 0x7fd22019ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd21fe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd220101110 0x7fd22019ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55568/0 (socket says 172.21.15.102:55568) 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd21fe00640 1 -- 172.21.15.102:0/3976961352 learned_addr learned my addr 172.21.15.102:0/3976961352 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd224a00640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd220102410 0x7fd2201a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.397+0000 7fd21f400640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd220101a70 0x7fd22019b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd21fe00640 1 -- 172.21.15.102:0/3976961352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd220102410 msgr2=0x7fd2201a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd21fe00640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd220102410 0x7fd2201a21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd21fe00640 1 -- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd220101a70 msgr2=0x7fd22019b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd21fe00640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd220101a70 0x7fd22019b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd21fe00640 1 -- 172.21.15.102:0/3976961352 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2201a2860 con 0x7fd220101110 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd21f400640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd220101a70 0x7fd22019b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd21fe00640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd220101110 0x7fd22019ac50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd21000e9c0 tx=0x7fd21000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd21cc00640 1 -- 172.21.15.102:0/3976961352 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21000ccc0 con 0x7fd220101110 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd225400640 1 -- 172.21.15.102:0/3976961352 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2201a2b50 con 0x7fd220101110 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd21cc00640 1 -- 172.21.15.102:0/3976961352 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2100054d0 con 0x7fd220101110 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd21cc00640 1 -- 172.21.15.102:0/3976961352 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210010430 con 0x7fd220101110 2024-04-26T20:34:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.398+0000 7fd225400640 1 -- 172.21.15.102:0/3976961352 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2201a30a0 con 0x7fd220101110 2024-04-26T20:34:30.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.399+0000 7fd225400640 1 -- 172.21.15.102:0/3976961352 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd22010c090 con 0x7fd220101110 2024-04-26T20:34:30.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.401+0000 7fd21cc00640 1 -- 172.21.15.102:0/3976961352 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fd210004f80 con 0x7fd220101110 2024-04-26T20:34:30.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.403+0000 7fd21cc00640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd2040a3c40 0x7fd2040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:30.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.403+0000 7fd21cc00640 1 -- 172.21.15.102:0/3976961352 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9179+0+0 (secure 0 0 0) 0x7fd2100d1230 con 0x7fd220101110 2024-04-26T20:34:30.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.403+0000 7fd21f400640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd2040a3c40 0x7fd2040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:30.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.404+0000 7fd21f400640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd2040a3c40 0x7fd2040a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd20c02a930 tx=0x7fd20c030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:30.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.405+0000 7fd21cc00640 1 -- 172.21.15.102:0/3976961352 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd21009d6a0 con 0x7fd220101110 2024-04-26T20:34:30.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.606+0000 7fd225400640 1 -- 172.21.15.102:0/3976961352 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd22002dd40 con 0x7fd220101110 2024-04-26T20:34:30.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.607+0000 7fd21cc00640 1 -- 172.21.15.102:0/3976961352 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7fd21009cfa0 con 0x7fd220101110 2024-04-26T20:34:30.608 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:34:30.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 -- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd2040a3c40 msgr2=0x7fd2040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd2040a3c40 0x7fd2040a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd20c02a930 tx=0x7fd20c030450 comp rx=0 tx=0).stop 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 -- 172.21.15.102:0/3976961352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd220101110 msgr2=0x7fd22019ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd220101110 0x7fd22019ac50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd21000e9c0 tx=0x7fd21000ee80 comp rx=0 tx=0).stop 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 -- 172.21.15.102:0/3976961352 shutdown_connections 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd2040a3c40 0x7fd2040a6100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd220101a70 0x7fd22019b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd220102410 0x7fd2201a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 --2- 172.21.15.102:0/3976961352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd220101110 0x7fd22019ac50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 -- 172.21.15.102:0/3976961352 >> 172.21.15.102:0/3976961352 conn(0x7fd2200fccd0 msgr2=0x7fd2200fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 -- 172.21.15.102:0/3976961352 shutdown_connections 2024-04-26T20:34:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.609+0000 7fd225400640 1 -- 172.21.15.102:0/3976961352 wait complete. 2024-04-26T20:34:30.629 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:34:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.814+0000 7f24f3200640 1 -- 172.21.15.102:0/1874692701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f4109f80 msgr2=0x7f24f410a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.814+0000 7f24f3200640 1 --2- 172.21.15.102:0/1874692701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f4109f80 0x7f24f410a400 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009900 tx=0x7f24dc02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.814+0000 7f24f3200640 1 -- 172.21.15.102:0/1874692701 shutdown_connections 2024-04-26T20:34:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.814+0000 7f24f3200640 1 --2- 172.21.15.102:0/1874692701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f4109f80 0x7f24f410a400 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.814+0000 7f24f3200640 1 --2- 172.21.15.102:0/1874692701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24f410a940 0x7f24f41111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.814+0000 7f24f3200640 1 --2- 172.21.15.102:0/1874692701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24f4108d90 0x7f24f41091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.814+0000 7f24f3200640 1 -- 172.21.15.102:0/1874692701 >> 172.21.15.102:0/1874692701 conn(0x7f24f4077d30 msgr2=0x7f24f407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:30.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.814+0000 7f24f3200640 1 -- 172.21.15.102:0/1874692701 shutdown_connections 2024-04-26T20:34:30.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.814+0000 7f24f3200640 1 -- 172.21.15.102:0/1874692701 wait complete. 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.815+0000 7f24f3200640 1 Processor -- start 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.815+0000 7f24f3200640 1 -- start start 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.815+0000 7f24f3200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24f4108d90 0x7f24f419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.815+0000 7f24f3200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24f4109f80 0x7f24f419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.815+0000 7f24f3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f410a940 0x7f24f41a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.815+0000 7f24f3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24f4114910 con 0x7f24f410a940 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.815+0000 7f24f3200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f24f4114790 con 0x7f24f4108d90 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.815+0000 7f24f3200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f24f4114a90 con 0x7f24f4109f80 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24f2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f410a940 0x7f24f41a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24f2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f410a940 0x7f24f41a6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50114/0 (socket says 172.21.15.102:50114) 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24e9400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24f4109f80 0x7f24f419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24f2800640 1 -- 172.21.15.102:0/2194088898 learned_addr learned my addr 172.21.15.102:0/2194088898 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24f1e00640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24f4108d90 0x7f24f419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24f2800640 1 -- 172.21.15.102:0/2194088898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24f4109f80 msgr2=0x7f24f419f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24f2800640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24f4109f80 0x7f24f419f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24f2800640 1 -- 172.21.15.102:0/2194088898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24f4108d90 msgr2=0x7f24f419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24f2800640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24f4108d90 0x7f24f419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24f2800640 1 -- 172.21.15.102:0/2194088898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f41a6bb0 con 0x7f24f410a940 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24f2800640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f410a940 0x7f24f41a6540 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00eaa0 tx=0x7f24ec00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.816+0000 7f24eb400640 1 -- 172.21.15.102:0/2194088898 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec00cdb0 con 0x7f24f410a940 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.817+0000 7f24eb400640 1 -- 172.21.15.102:0/2194088898 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24ec005020 con 0x7f24f410a940 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.817+0000 7f24eb400640 1 -- 172.21.15.102:0/2194088898 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec0116d0 con 0x7f24f410a940 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.817+0000 7f24f3200640 1 -- 172.21.15.102:0/2194088898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f41a6ea0 con 0x7f24f410a940 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.817+0000 7f24f3200640 1 -- 172.21.15.102:0/2194088898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f41a7360 con 0x7f24f410a940 2024-04-26T20:34:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.818+0000 7f24f3200640 1 -- 172.21.15.102:0/2194088898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f41100b0 con 0x7f24f410a940 2024-04-26T20:34:30.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.820+0000 7f24eb400640 1 -- 172.21.15.102:0/2194088898 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f24ec0051d0 con 0x7f24f410a940 2024-04-26T20:34:30.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.821+0000 7f24eb400640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f24d40a3c40 0x7f24d40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:30.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.821+0000 7f24eb400640 1 -- 172.21.15.102:0/2194088898 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f24ec0d0a40 con 0x7f24f410a940 2024-04-26T20:34:30.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.821+0000 7f24f1e00640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f24d40a3c40 0x7f24d40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:30.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.822+0000 7f24f1e00640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f24d40a3c40 0x7f24d40a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f24e4005fd0 tx=0x7f24e400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:30.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:30.825+0000 7f24eb400640 1 -- 172.21.15.102:0/2194088898 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f24ec016030 con 0x7f24f410a940 2024-04-26T20:34:30.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-26T20:34:30.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-26T20:34:30.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-26T20:34:30.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:30 smithi102 ceph-mon[28094]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T20:34:30.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:30 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2879515681' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:34:30.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:30 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1442170259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:34:30.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:30 smithi102 ceph-mon[28094]: pgmap v383: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-26T20:34:30.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:30 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3976961352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:34:30.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:30 smithi102 ceph-mon[28094]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T20:34:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-26T20:34:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-26T20:34:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-26T20:34:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:30 smithi112 ceph-mon[32769]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T20:34:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:30 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2879515681' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:34:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:30 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1442170259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:34:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:30 smithi112 ceph-mon[32769]: pgmap v383: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-26T20:34:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:30 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3976961352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:34:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:30 smithi112 ceph-mon[32769]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T20:34:31.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.050+0000 7f24f3200640 1 -- 172.21.15.102:0/2194088898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24f402dad0 con 0x7f24f410a940 2024-04-26T20:34:31.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.051+0000 7f24eb400640 1 -- 172.21.15.102:0/2194088898 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11954 (secure 0 0 0) 0x7f24ec09c860 con 0x7f24f410a940 2024-04-26T20:34:31.053 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:31.053 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:32:25.769708+0000","modified":"2024-04-26T20:32:55.699799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24593,"mds_1":24565,"mds_2":24547,"mds_3":24533,"mds_4":24526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24565,24593,24547,24533,24526]},"id":1}]} 2024-04-26T20:34:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-26T20:34:31.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.054+0000 7f24f3200640 1 -- 172.21.15.102:0/2194088898 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f24d40a3c40 msgr2=0x7f24d40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.054+0000 7f24f3200640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f24d40a3c40 0x7f24d40a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f24e4005fd0 tx=0x7f24e400b410 comp rx=0 tx=0).stop 2024-04-26T20:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.054+0000 7f24f3200640 1 -- 172.21.15.102:0/2194088898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f410a940 msgr2=0x7f24f41a6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.054+0000 7f24f3200640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f410a940 0x7f24f41a6540 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00eaa0 tx=0x7f24ec00ef60 comp rx=0 tx=0).stop 2024-04-26T20:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.054+0000 7f24f3200640 1 -- 172.21.15.102:0/2194088898 shutdown_connections 2024-04-26T20:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.054+0000 7f24f3200640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f24d40a3c40 0x7f24d40a6100 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.054+0000 7f24f3200640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f410a940 0x7f24f41a6540 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.054+0000 7f24f3200640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24f4109f80 0x7f24f419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.054+0000 7f24f3200640 1 --2- 172.21.15.102:0/2194088898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24f4108d90 0x7f24f419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.054+0000 7f24f3200640 1 -- 172.21.15.102:0/2194088898 >> 172.21.15.102:0/2194088898 conn(0x7f24f4077d30 msgr2=0x7f24f410d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.055+0000 7f24f3200640 1 -- 172.21.15.102:0/2194088898 shutdown_connections 2024-04-26T20:34:31.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.055+0000 7f24f3200640 1 -- 172.21.15.102:0/2194088898 wait complete. 2024-04-26T20:34:31.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:34:31.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-26T20:34:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-26T20:34:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-26T20:34:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:30 smithi130 ceph-mon[32704]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T20:34:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:30 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2879515681' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:34:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:30 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1442170259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:34:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:30 smithi130 ceph-mon[32704]: pgmap v383: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-26T20:34:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:30 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3976961352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:34:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:30 smithi130 ceph-mon[32704]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T20:34:31.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.273+0000 7efd04c00640 1 -- 172.21.15.102:0/4053410954 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd0010a830 msgr2=0x7efd0010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.273+0000 7efd04c00640 1 --2- 172.21.15.102:0/4053410954 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd0010a830 0x7efd0010ccc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009ca0 tx=0x7efcf802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.273+0000 7efd04c00640 1 -- 172.21.15.102:0/4053410954 shutdown_connections 2024-04-26T20:34:31.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.273+0000 7efd04c00640 1 --2- 172.21.15.102:0/4053410954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd00075390 0x7efd00075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.273+0000 7efd04c00640 1 --2- 172.21.15.102:0/4053410954 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd0010a830 0x7efd0010ccc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.273+0000 7efd04c00640 1 --2- 172.21.15.102:0/4053410954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd00076970 0x7efd00074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.273+0000 7efd04c00640 1 -- 172.21.15.102:0/4053410954 >> 172.21.15.102:0/4053410954 conn(0x7efd000fdee0 msgr2=0x7efd00100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:31.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.273+0000 7efd04c00640 1 -- 172.21.15.102:0/4053410954 shutdown_connections 2024-04-26T20:34:31.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.273+0000 7efd04c00640 1 -- 172.21.15.102:0/4053410954 wait complete. 2024-04-26T20:34:31.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.274+0000 7efd04c00640 1 Processor -- start 2024-04-26T20:34:31.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efd04c00640 1 -- start start 2024-04-26T20:34:31.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efd04c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd00075390 0x7efd0019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:31.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efd04c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd00076970 0x7efd0019b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:31.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efd04c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd0010a830 0x7efd001a2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:31.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efd04c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efd001106d0 con 0x7efd0010a830 2024-04-26T20:34:31.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efd04c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efd00110550 con 0x7efd00075390 2024-04-26T20:34:31.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efd04c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7efd00110850 con 0x7efd00076970 2024-04-26T20:34:31.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efcfe800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd0010a830 0x7efd001a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:31.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efcfde00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd00075390 0x7efd0019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efcfde00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd00075390 0x7efd0019ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:40514/0 (socket says 172.21.15.102:40514) 2024-04-26T20:34:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efcfde00640 1 -- 172.21.15.102:0/1710174069 learned_addr learned my addr 172.21.15.102:0/1710174069 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.275+0000 7efcfd400640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd00076970 0x7efd0019b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.276+0000 7efcfde00640 1 -- 172.21.15.102:0/1710174069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd00076970 msgr2=0x7efd0019b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.276+0000 7efcfde00640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd00076970 0x7efd0019b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.276+0000 7efcfde00640 1 -- 172.21.15.102:0/1710174069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd0010a830 msgr2=0x7efd001a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.276+0000 7efcfde00640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd0010a830 0x7efd001a2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.276+0000 7efcfde00640 1 -- 172.21.15.102:0/1710174069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd001a2910 con 0x7efd00075390 2024-04-26T20:34:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.276+0000 7efcfde00640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd00075390 0x7efd0019ad00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efcf000e9f0 tx=0x7efcf000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:31.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.276+0000 7efcf6a00640 1 -- 172.21.15.102:0/1710174069 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf000cd80 con 0x7efd00075390 2024-04-26T20:34:31.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.276+0000 7efcf6a00640 1 -- 172.21.15.102:0/1710174069 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcf00054d0 con 0x7efd00075390 2024-04-26T20:34:31.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.276+0000 7efcf6a00640 1 -- 172.21.15.102:0/1710174069 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0010690 con 0x7efd00075390 2024-04-26T20:34:31.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.276+0000 7efd04c00640 1 -- 172.21.15.102:0/1710174069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd001a2c00 con 0x7efd00075390 2024-04-26T20:34:31.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.277+0000 7efd04c00640 1 -- 172.21.15.102:0/1710174069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd001a3100 con 0x7efd00075390 2024-04-26T20:34:31.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.278+0000 7efd04c00640 1 -- 172.21.15.102:0/1710174069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd00109560 con 0x7efd00075390 2024-04-26T20:34:31.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.280+0000 7efcf6a00640 1 -- 172.21.15.102:0/1710174069 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7efcf0004e90 con 0x7efd00075390 2024-04-26T20:34:31.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.281+0000 7efcf6a00640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efce40a3c40 0x7efce40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:31.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.281+0000 7efcf6a00640 1 -- 172.21.15.102:0/1710174069 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7efcf00d0f60 con 0x7efd00075390 2024-04-26T20:34:31.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.281+0000 7efcfd400640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efce40a3c40 0x7efce40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:31.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.282+0000 7efcfd400640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efce40a3c40 0x7efce40a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efcec009bd0 tx=0x7efcec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.283+0000 7efcf6a00640 1 -- 172.21.15.102:0/1710174069 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efcf009d3d0 con 0x7efd00075390 2024-04-26T20:34:31.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.489+0000 7efd04c00640 1 -- 172.21.15.102:0/1710174069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd0002dd20 con 0x7efd00075390 2024-04-26T20:34:31.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.491+0000 7efcf6a00640 1 -- 172.21.15.102:0/1710174069 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24833 (secure 0 0 0) 0x7efcf009ccd0 con 0x7efd00075390 2024-04-26T20:34:31.492 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:31.492 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:34:30.697401+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:34:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 -- 172.21.15.102:0/1710174069 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efce40a3c40 msgr2=0x7efce40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efce40a3c40 0x7efce40a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efcec009bd0 tx=0x7efcec005f90 comp rx=0 tx=0).stop 2024-04-26T20:34:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 -- 172.21.15.102:0/1710174069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd00075390 msgr2=0x7efd0019ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd00075390 0x7efd0019ad00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efcf000e9f0 tx=0x7efcf000edf0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 -- 172.21.15.102:0/1710174069 shutdown_connections 2024-04-26T20:34:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efce40a3c40 0x7efce40a6100 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd0010a830 0x7efd001a2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd00076970 0x7efd0019b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 --2- 172.21.15.102:0/1710174069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd00075390 0x7efd0019ad00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 -- 172.21.15.102:0/1710174069 >> 172.21.15.102:0/1710174069 conn(0x7efd000fdee0 msgr2=0x7efd00106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:31.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 -- 172.21.15.102:0/1710174069 shutdown_connections 2024-04-26T20:34:31.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.494+0000 7efd04c00640 1 -- 172.21.15.102:0/1710174069 wait complete. 2024-04-26T20:34:31.515 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:27:10.527933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:32:23.439237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:32:24.702263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:32:28.348785+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:34:31.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:34:31.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.723+0000 7f7e77e00640 1 -- 172.21.15.102:0/946717135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e78104a70 msgr2=0x7f7e78104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.723+0000 7f7e77e00640 1 --2- 172.21.15.102:0/946717135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e78104a70 0x7f7e78104e90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009860 tx=0x7f7e6802a340 comp rx=0 tx=0).stop 2024-04-26T20:34:31.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.723+0000 7f7e77e00640 1 -- 172.21.15.102:0/946717135 shutdown_connections 2024-04-26T20:34:31.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.723+0000 7f7e77e00640 1 --2- 172.21.15.102:0/946717135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e78106620 0x7f7e7810ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.723+0000 7f7e77e00640 1 --2- 172.21.15.102:0/946717135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e78105c60 0x7f7e781060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.723+0000 7f7e77e00640 1 --2- 172.21.15.102:0/946717135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e78104a70 0x7f7e78104e90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.723+0000 7f7e77e00640 1 -- 172.21.15.102:0/946717135 >> 172.21.15.102:0/946717135 conn(0x7f7e780fffe0 msgr2=0x7f7e78102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:31.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.723+0000 7f7e77e00640 1 -- 172.21.15.102:0/946717135 shutdown_connections 2024-04-26T20:34:31.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.723+0000 7f7e77e00640 1 -- 172.21.15.102:0/946717135 wait complete. 2024-04-26T20:34:31.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.724+0000 7f7e77e00640 1 Processor -- start 2024-04-26T20:34:31.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.724+0000 7f7e77e00640 1 -- start start 2024-04-26T20:34:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e77e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e78104a70 0x7f7e7819aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e77e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e78105c60 0x7f7e7819b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e77e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e78106620 0x7f7e781a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e77e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7e78110570 con 0x7f7e78106620 2024-04-26T20:34:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e77e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7e781103f0 con 0x7f7e78104a70 2024-04-26T20:34:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e77e00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7e781106f0 con 0x7f7e78105c60 2024-04-26T20:34:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e76000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e78105c60 0x7f7e7819b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e76000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e78105c60 0x7f7e7819b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55634/0 (socket says 172.21.15.102:55634) 2024-04-26T20:34:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e76000640 1 -- 172.21.15.102:0/3066843382 learned_addr learned my addr 172.21.15.102:0/3066843382 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e77400640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e78106620 0x7f7e781a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.725+0000 7f7e76a00640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e78104a70 0x7f7e7819aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e76000640 1 -- 172.21.15.102:0/3066843382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e78104a70 msgr2=0x7f7e7819aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e76000640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e78104a70 0x7f7e7819aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e76000640 1 -- 172.21.15.102:0/3066843382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e78106620 msgr2=0x7f7e781a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e76000640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e78106620 0x7f7e781a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e76000640 1 -- 172.21.15.102:0/3066843382 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e781a28e0 con 0x7f7e78105c60 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e76000640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e78105c60 0x7f7e7819b1e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e9c0 tx=0x7f7e6400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e7ca00640 1 -- 172.21.15.102:0/3066843382 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400ccc0 con 0x7f7e78105c60 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e7ca00640 1 -- 172.21.15.102:0/3066843382 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e640054d0 con 0x7f7e78105c60 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e77e00640 1 -- 172.21.15.102:0/3066843382 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e781a2bd0 con 0x7f7e78105c60 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e7ca00640 1 -- 172.21.15.102:0/3066843382 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64010430 con 0x7f7e78105c60 2024-04-26T20:34:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.726+0000 7f7e77e00640 1 -- 172.21.15.102:0/3066843382 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e781a3120 con 0x7f7e78105c60 2024-04-26T20:34:31.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.728+0000 7f7e77e00640 1 -- 172.21.15.102:0/3066843382 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e7810be60 con 0x7f7e78105c60 2024-04-26T20:34:31.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.730+0000 7f7e7ca00640 1 -- 172.21.15.102:0/3066843382 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f7e64004f80 con 0x7f7e78105c60 2024-04-26T20:34:31.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.731+0000 7f7e7ca00640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7e5c0a3c40 0x7f7e5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:31.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.731+0000 7f7e7ca00640 1 -- 172.21.15.102:0/3066843382 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f7e640d0d70 con 0x7f7e78105c60 2024-04-26T20:34:31.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.732+0000 7f7e76a00640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7e5c0a3c40 0x7f7e5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:31.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.732+0000 7f7e76a00640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7e5c0a3c40 0x7f7e5c0a6100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7e7819bb70 tx=0x7f7e68030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:31.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.735+0000 7f7e7ca00640 1 -- 172.21.15.102:0/3066843382 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7e6409d1e0 con 0x7f7e78105c60 2024-04-26T20:34:31.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:31 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2194088898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:34:31.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:31 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1710174069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:34:31.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.938+0000 7f7e77e00640 1 -- 172.21.15.102:0/3066843382 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7e7802dc80 con 0x7f7e78105c60 2024-04-26T20:34:31.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.939+0000 7f7e7ca00640 1 -- 172.21.15.102:0/3066843382 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f7e6409cae0 con 0x7f7e78105c60 2024-04-26T20:34:31.940 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-26T20:34:31.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.941+0000 7f7e77e00640 1 -- 172.21.15.102:0/3066843382 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7e5c0a3c40 msgr2=0x7f7e5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.941+0000 7f7e77e00640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7e5c0a3c40 0x7f7e5c0a6100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7e7819bb70 tx=0x7f7e68030450 comp rx=0 tx=0).stop 2024-04-26T20:34:31.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.941+0000 7f7e77e00640 1 -- 172.21.15.102:0/3066843382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e78105c60 msgr2=0x7f7e7819b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:31.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.941+0000 7f7e77e00640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e78105c60 0x7f7e7819b1e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e9c0 tx=0x7f7e6400ee80 comp rx=0 tx=0).stop 2024-04-26T20:34:31.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.942+0000 7f7e77e00640 1 -- 172.21.15.102:0/3066843382 shutdown_connections 2024-04-26T20:34:31.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.942+0000 7f7e77e00640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7e5c0a3c40 0x7f7e5c0a6100 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.942+0000 7f7e77e00640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e78106620 0x7f7e781a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.942+0000 7f7e77e00640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e78105c60 0x7f7e7819b1e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.942+0000 7f7e77e00640 1 --2- 172.21.15.102:0/3066843382 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e78104a70 0x7f7e7819aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:31.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.942+0000 7f7e77e00640 1 -- 172.21.15.102:0/3066843382 >> 172.21.15.102:0/3066843382 conn(0x7f7e780fffe0 msgr2=0x7f7e78109030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:31.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.942+0000 7f7e77e00640 1 -- 172.21.15.102:0/3066843382 shutdown_connections 2024-04-26T20:34:31.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:31.942+0000 7f7e77e00640 1 -- 172.21.15.102:0/3066843382 wait complete. 2024-04-26T20:34:31.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:34:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:31 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2194088898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:34:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:31 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1710174069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:34:32.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:31 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2194088898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:34:32.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:31 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1710174069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:34:32.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.165+0000 7ff837200640 1 -- 172.21.15.102:0/2168844308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff830108520 msgr2=0x7ff83010a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.165+0000 7ff837200640 1 --2- 172.21.15.102:0/2168844308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff830108520 0x7ff83010a970 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009900 tx=0x7ff82c02a410 comp rx=0 tx=0).stop 2024-04-26T20:34:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.165+0000 7ff837200640 1 -- 172.21.15.102:0/2168844308 shutdown_connections 2024-04-26T20:34:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.165+0000 7ff837200640 1 --2- 172.21.15.102:0/2168844308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff830108520 0x7ff83010a970 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.165+0000 7ff837200640 1 --2- 172.21.15.102:0/2168844308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff830105bf0 0x7ff830107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.165+0000 7ff837200640 1 --2- 172.21.15.102:0/2168844308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8301032c0 0x7ff8301056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.165+0000 7ff837200640 1 -- 172.21.15.102:0/2168844308 >> 172.21.15.102:0/2168844308 conn(0x7ff8300fccd0 msgr2=0x7ff8300ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.165+0000 7ff837200640 1 -- 172.21.15.102:0/2168844308 shutdown_connections 2024-04-26T20:34:32.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.165+0000 7ff837200640 1 -- 172.21.15.102:0/2168844308 wait complete. 2024-04-26T20:34:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.166+0000 7ff837200640 1 Processor -- start 2024-04-26T20:34:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.166+0000 7ff837200640 1 -- start start 2024-04-26T20:34:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff837200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8301032c0 0x7ff830198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff837200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff830105bf0 0x7ff830198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff837200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff830108520 0x7ff8301a0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff837200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff83010e3e0 con 0x7ff830105bf0 2024-04-26T20:34:32.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff837200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff83010e260 con 0x7ff830108520 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff837200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff83010e560 con 0x7ff8301032c0 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff835400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff830105bf0 0x7ff830198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff835400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff830105bf0 0x7ff830198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50174/0 (socket says 172.21.15.102:50174) 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff835400640 1 -- 172.21.15.102:0/1772845124 learned_addr learned my addr 172.21.15.102:0/1772845124 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff835e00640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8301032c0 0x7ff830198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.167+0000 7ff836800640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff830108520 0x7ff8301a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff835400640 1 -- 172.21.15.102:0/1772845124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8301032c0 msgr2=0x7ff830198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff835400640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8301032c0 0x7ff830198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff835400640 1 -- 172.21.15.102:0/1772845124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff830108520 msgr2=0x7ff8301a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff835400640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff830108520 0x7ff8301a0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff835400640 1 -- 172.21.15.102:0/1772845124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8301a0670 con 0x7ff830105bf0 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff836800640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff830108520 0x7ff8301a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff835400640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff830105bf0 0x7ff830198f90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff82000e9a0 tx=0x7ff82000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff826a00640 1 -- 172.21.15.102:0/1772845124 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82000cd80 con 0x7ff830105bf0 2024-04-26T20:34:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff826a00640 1 -- 172.21.15.102:0/1772845124 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8200035f0 con 0x7ff830105bf0 2024-04-26T20:34:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff837200640 1 -- 172.21.15.102:0/1772845124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8301a0960 con 0x7ff830105bf0 2024-04-26T20:34:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff826a00640 1 -- 172.21.15.102:0/1772845124 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff820010690 con 0x7ff830105bf0 2024-04-26T20:34:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.168+0000 7ff837200640 1 -- 172.21.15.102:0/1772845124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8301a0f00 con 0x7ff830105bf0 2024-04-26T20:34:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.169+0000 7ff837200640 1 -- 172.21.15.102:0/1772845124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8300fe7c0 con 0x7ff830105bf0 2024-04-26T20:34:32.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.171+0000 7ff826a00640 1 -- 172.21.15.102:0/1772845124 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff820004000 con 0x7ff830105bf0 2024-04-26T20:34:32.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.175+0000 7ff826a00640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff81c0a3cc0 0x7ff81c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:32.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.175+0000 7ff826a00640 1 -- 172.21.15.102:0/1772845124 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7ff8200d0b30 con 0x7ff830105bf0 2024-04-26T20:34:32.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.175+0000 7ff835e00640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff81c0a3cc0 0x7ff81c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:32.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.176+0000 7ff835e00640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff81c0a3cc0 0x7ff81c0a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff828005fd0 tx=0x7ff828009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:32.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.179+0000 7ff826a00640 1 -- 172.21.15.102:0/1772845124 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff8200042f0 con 0x7ff830105bf0 2024-04-26T20:34:32.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.380+0000 7ff837200640 1 -- 172.21.15.102:0/1772845124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff83002dcf0 con 0x7ff830105bf0 2024-04-26T20:34:32.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.381+0000 7ff826a00640 1 -- 172.21.15.102:0/1772845124 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7ff82009d000 con 0x7ff830105bf0 2024-04-26T20:34:32.383 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:34:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.384+0000 7ff837200640 1 -- 172.21.15.102:0/1772845124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff81c0a3cc0 msgr2=0x7ff81c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.384+0000 7ff837200640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff81c0a3cc0 0x7ff81c0a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff828005fd0 tx=0x7ff828009500 comp rx=0 tx=0).stop 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.384+0000 7ff837200640 1 -- 172.21.15.102:0/1772845124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff830105bf0 msgr2=0x7ff830198f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.384+0000 7ff837200640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff830105bf0 0x7ff830198f90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff82000e9a0 tx=0x7ff82000ee60 comp rx=0 tx=0).stop 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.384+0000 7ff837200640 1 -- 172.21.15.102:0/1772845124 shutdown_connections 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.384+0000 7ff837200640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff81c0a3cc0 0x7ff81c0a6180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.384+0000 7ff837200640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff830105bf0 0x7ff830198f90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.384+0000 7ff837200640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff830108520 0x7ff8301a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.384+0000 7ff837200640 1 --2- 172.21.15.102:0/1772845124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8301032c0 0x7ff830198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.384+0000 7ff837200640 1 -- 172.21.15.102:0/1772845124 >> 172.21.15.102:0/1772845124 conn(0x7ff8300fccd0 msgr2=0x7ff830103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.385+0000 7ff837200640 1 -- 172.21.15.102:0/1772845124 shutdown_connections 2024-04-26T20:34:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.385+0000 7ff837200640 1 -- 172.21.15.102:0/1772845124 wait complete. 2024-04-26T20:34:32.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:34:32.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.608+0000 7f55b9600640 1 -- 172.21.15.102:0/2430569649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b4109f80 msgr2=0x7f55b410a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:32.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.608+0000 7f55b9600640 1 --2- 172.21.15.102:0/2430569649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b4109f80 0x7f55b410a400 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f559c009900 tx=0x7f559c02a410 comp rx=0 tx=0).stop 2024-04-26T20:34:32.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.608+0000 7f55b9600640 1 -- 172.21.15.102:0/2430569649 shutdown_connections 2024-04-26T20:34:32.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.608+0000 7f55b9600640 1 --2- 172.21.15.102:0/2430569649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b4109f80 0x7f55b410a400 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.608+0000 7f55b9600640 1 --2- 172.21.15.102:0/2430569649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55b410a940 0x7f55b41111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.608+0000 7f55b9600640 1 --2- 172.21.15.102:0/2430569649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55b4108d90 0x7f55b41091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.608+0000 7f55b9600640 1 -- 172.21.15.102:0/2430569649 >> 172.21.15.102:0/2430569649 conn(0x7f55b4077d30 msgr2=0x7f55b407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:32.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.608+0000 7f55b9600640 1 -- 172.21.15.102:0/2430569649 shutdown_connections 2024-04-26T20:34:32.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.608+0000 7f55b9600640 1 -- 172.21.15.102:0/2430569649 wait complete. 2024-04-26T20:34:32.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.609+0000 7f55b9600640 1 Processor -- start 2024-04-26T20:34:32.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.609+0000 7f55b9600640 1 -- start start 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55b9600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b4108d90 0x7f55b419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55b9600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55b4109f80 0x7f55b419f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55b9600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55b410a940 0x7f55b41a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55b9600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f55b4114910 con 0x7f55b4108d90 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55b9600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55b4114790 con 0x7f55b4109f80 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55b9600640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f55b4114a90 con 0x7f55b410a940 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b4108d90 0x7f55b419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b4108d90 0x7f55b419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50184/0 (socket says 172.21.15.102:50184) 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55b3e00640 1 -- 172.21.15.102:0/1466409124 learned_addr learned my addr 172.21.15.102:0/1466409124 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55ab400640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55b4109f80 0x7f55b419f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.610+0000 7f55b8c00640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55b410a940 0x7f55b41a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b3e00640 1 -- 172.21.15.102:0/1466409124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55b410a940 msgr2=0x7f55b41a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b3e00640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55b410a940 0x7f55b41a6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b3e00640 1 -- 172.21.15.102:0/1466409124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55b4109f80 msgr2=0x7f55b419f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b3e00640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55b4109f80 0x7f55b419f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b3e00640 1 -- 172.21.15.102:0/1466409124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b41a6bf0 con 0x7f55b4108d90 2024-04-26T20:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b3e00640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b4108d90 0x7f55b419efd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f55a400e9c0 tx=0x7f55a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b1600640 1 -- 172.21.15.102:0/1466409124 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a400cd80 con 0x7f55b4108d90 2024-04-26T20:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b1600640 1 -- 172.21.15.102:0/1466409124 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55a40035f0 con 0x7f55b4108d90 2024-04-26T20:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b9600640 1 -- 172.21.15.102:0/1466409124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b41a6ee0 con 0x7f55b4108d90 2024-04-26T20:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b1600640 1 -- 172.21.15.102:0/1466409124 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a4010690 con 0x7f55b4108d90 2024-04-26T20:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.611+0000 7f55b9600640 1 -- 172.21.15.102:0/1466409124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b41a72f0 con 0x7f55b4108d90 2024-04-26T20:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.612+0000 7f55b9600640 1 -- 172.21.15.102:0/1466409124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b4110180 con 0x7f55b4108d90 2024-04-26T20:34:32.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.614+0000 7f55b1600640 1 -- 172.21.15.102:0/1466409124 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f55a4004000 con 0x7f55b4108d90 2024-04-26T20:34:32.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.616+0000 7f55b1600640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55980a3c40 0x7f55980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:32.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.616+0000 7f55b1600640 1 -- 172.21.15.102:0/1466409124 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f55a40d0e80 con 0x7f55b4108d90 2024-04-26T20:34:32.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.616+0000 7f55ab400640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55980a3c40 0x7f55980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:32.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.617+0000 7f55ab400640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55980a3c40 0x7f55980a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f559c031500 tx=0x7f559c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:32.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.620+0000 7f55b1600640 1 -- 172.21.15.102:0/1466409124 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f55a409d2f0 con 0x7f55b4108d90 2024-04-26T20:34:32.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.828+0000 7f55b9600640 1 -- 172.21.15.102:0/1466409124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f55b402dd80 con 0x7f55b4108d90 2024-04-26T20:34:32.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.829+0000 7f55b1600640 1 -- 172.21.15.102:0/1466409124 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f55a409cbf0 con 0x7f55b4108d90 2024-04-26T20:34:32.830 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-26T20:34:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.831+0000 7f55b9600640 1 -- 172.21.15.102:0/1466409124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55980a3c40 msgr2=0x7f55980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.831+0000 7f55b9600640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55980a3c40 0x7f55980a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f559c031500 tx=0x7f559c005a20 comp rx=0 tx=0).stop 2024-04-26T20:34:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.831+0000 7f55b9600640 1 -- 172.21.15.102:0/1466409124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b4108d90 msgr2=0x7f55b419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.831+0000 7f55b9600640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b4108d90 0x7f55b419efd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f55a400e9c0 tx=0x7f55a400ee80 comp rx=0 tx=0).stop 2024-04-26T20:34:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.832+0000 7f55b9600640 1 -- 172.21.15.102:0/1466409124 shutdown_connections 2024-04-26T20:34:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.832+0000 7f55b9600640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55980a3c40 0x7f55980a6100 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.832+0000 7f55b9600640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55b410a940 0x7f55b41a6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.832+0000 7f55b9600640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55b4109f80 0x7f55b419f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.832+0000 7f55b9600640 1 --2- 172.21.15.102:0/1466409124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b4108d90 0x7f55b419efd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:32.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.832+0000 7f55b9600640 1 -- 172.21.15.102:0/1466409124 >> 172.21.15.102:0/1466409124 conn(0x7f55b4077d30 msgr2=0x7f55b410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:32.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.832+0000 7f55b9600640 1 -- 172.21.15.102:0/1466409124 shutdown_connections 2024-04-26T20:34:32.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:32.832+0000 7f55b9600640 1 -- 172.21.15.102:0/1466409124 wait complete. 2024-04-26T20:34:32.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:34:33.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:32 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3066843382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:34:33.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:32 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1772845124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:34:33.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:32 smithi112 ceph-mon[32769]: pgmap v385: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T20:34:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.057+0000 7fc6fee00640 1 -- 172.21.15.102:0/3663782841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6f81043f0 msgr2=0x7fc6f810ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.057+0000 7fc6fee00640 1 --2- 172.21.15.102:0/3663782841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6f81043f0 0x7fc6f810ac80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009be0 tx=0x7fc6f002a6d0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.057+0000 7fc6fee00640 1 -- 172.21.15.102:0/3663782841 shutdown_connections 2024-04-26T20:34:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.057+0000 7fc6fee00640 1 --2- 172.21.15.102:0/3663782841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6f81043f0 0x7fc6f810ac80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.057+0000 7fc6fee00640 1 --2- 172.21.15.102:0/3663782841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6f8103a30 0x7fc6f8103eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.057+0000 7fc6fee00640 1 --2- 172.21.15.102:0/3663782841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f8102840 0x7fc6f8102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.057+0000 7fc6fee00640 1 -- 172.21.15.102:0/3663782841 >> 172.21.15.102:0/3663782841 conn(0x7fc6f80fddd0 msgr2=0x7fc6f81001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:33.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.057+0000 7fc6fee00640 1 -- 172.21.15.102:0/3663782841 shutdown_connections 2024-04-26T20:34:33.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.057+0000 7fc6fee00640 1 -- 172.21.15.102:0/3663782841 wait complete. 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.058+0000 7fc6fee00640 1 Processor -- start 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.058+0000 7fc6fee00640 1 -- start start 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.058+0000 7fc6fee00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6f8102840 0x7fc6f819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.058+0000 7fc6fee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f8103a30 0x7fc6f819b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.058+0000 7fc6fee00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6f81043f0 0x7fc6f81a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.058+0000 7fc6fee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f810e370 con 0x7fc6f8103a30 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.058+0000 7fc6fee00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f810e1f0 con 0x7fc6f81043f0 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.058+0000 7fc6fee00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f810e4f0 con 0x7fc6f8102840 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f8103a30 0x7fc6f819b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fe400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6f81043f0 0x7fc6f81a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f8103a30 0x7fc6f819b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50202/0 (socket says 172.21.15.102:50202) 2024-04-26T20:34:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fd000640 1 -- 172.21.15.102:0/3253928532 learned_addr learned my addr 172.21.15.102:0/3253928532 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fda00640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6f8102840 0x7fc6f819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fe400640 1 -- 172.21.15.102:0/3253928532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6f8102840 msgr2=0x7fc6f819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fe400640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6f8102840 0x7fc6f819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fe400640 1 -- 172.21.15.102:0/3253928532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f8103a30 msgr2=0x7fc6f819b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fe400640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f8103a30 0x7fc6f819b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fe400640 1 -- 172.21.15.102:0/3253928532 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f81a28c0 con 0x7fc6f81043f0 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fd000640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f8103a30 0x7fc6f819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.059+0000 7fc6fe400640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6f81043f0 0x7fc6f81a2220 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009b30 tx=0x7fc6f0030d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.060+0000 7fc6f6000640 1 -- 172.21.15.102:0/3253928532 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f0041020 con 0x7fc6f81043f0 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.060+0000 7fc6f6000640 1 -- 172.21.15.102:0/3253928532 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6f00318f0 con 0x7fc6f81043f0 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.060+0000 7fc6fee00640 1 -- 172.21.15.102:0/3253928532 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6f81a2b50 con 0x7fc6f81043f0 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.060+0000 7fc6f6000640 1 -- 172.21.15.102:0/3253928532 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f0036040 con 0x7fc6f81043f0 2024-04-26T20:34:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.060+0000 7fc6fee00640 1 -- 172.21.15.102:0/3253928532 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6f81a3010 con 0x7fc6f81043f0 2024-04-26T20:34:33.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.061+0000 7fc6fee00640 1 -- 172.21.15.102:0/3253928532 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6f81a31e0 con 0x7fc6f81043f0 2024-04-26T20:34:33.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.063+0000 7fc6f6000640 1 -- 172.21.15.102:0/3253928532 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc6f0044020 con 0x7fc6f81043f0 2024-04-26T20:34:33.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.064+0000 7fc6f6000640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc6e00a3c40 0x7fc6e00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.065+0000 7fc6f6000640 1 -- 172.21.15.102:0/3253928532 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc6f00f18d0 con 0x7fc6f81043f0 2024-04-26T20:34:33.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.065+0000 7fc6fda00640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc6e00a3c40 0x7fc6e00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.065+0000 7fc6fda00640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc6e00a3c40 0x7fc6e00a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6f819bb50 tx=0x7fc6ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:33.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.067+0000 7fc6f6000640 1 -- 172.21.15.102:0/3253928532 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc6f004b020 con 0x7fc6f81043f0 2024-04-26T20:34:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:32 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3066843382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:34:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:32 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1772845124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:34:33.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:32 smithi130 ceph-mon[32704]: pgmap v385: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T20:34:33.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:32 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3066843382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:34:33.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:32 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1772845124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:34:33.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:32 smithi102 ceph-mon[28094]: pgmap v385: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T20:34:33.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.267+0000 7fc6fee00640 1 -- 172.21.15.102:0/3253928532 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc6f81a3570 con 0x7fc6f81043f0 2024-04-26T20:34:33.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.268+0000 7fc6f6000640 1 -- 172.21.15.102:0/3253928532 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fc6f00bde90 con 0x7fc6f81043f0 2024-04-26T20:34:33.269 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:34:33.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.271+0000 7fc6fee00640 1 -- 172.21.15.102:0/3253928532 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc6e00a3c40 msgr2=0x7fc6e00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.271+0000 7fc6fee00640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc6e00a3c40 0x7fc6e00a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6f819bb50 tx=0x7fc6ec009260 comp rx=0 tx=0).stop 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.271+0000 7fc6fee00640 1 -- 172.21.15.102:0/3253928532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6f81043f0 msgr2=0x7fc6f81a2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.271+0000 7fc6fee00640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6f81043f0 0x7fc6f81a2220 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009b30 tx=0x7fc6f0030d50 comp rx=0 tx=0).stop 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.271+0000 7fc6fee00640 1 -- 172.21.15.102:0/3253928532 shutdown_connections 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.271+0000 7fc6fee00640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc6e00a3c40 0x7fc6e00a6100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.271+0000 7fc6fee00640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f8103a30 0x7fc6f819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.271+0000 7fc6fee00640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6f81043f0 0x7fc6f81a2220 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.271+0000 7fc6fee00640 1 --2- 172.21.15.102:0/3253928532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6f8102840 0x7fc6f819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.271+0000 7fc6fee00640 1 -- 172.21.15.102:0/3253928532 >> 172.21.15.102:0/3253928532 conn(0x7fc6f80fddd0 msgr2=0x7fc6f8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.272+0000 7fc6fee00640 1 -- 172.21.15.102:0/3253928532 shutdown_connections 2024-04-26T20:34:33.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.272+0000 7fc6fee00640 1 -- 172.21.15.102:0/3253928532 wait complete. 2024-04-26T20:34:33.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:34:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.490+0000 7f92afe00640 1 -- 172.21.15.102:0/3488193886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f92b0106540 msgr2=0x7f92b010cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.490+0000 7f92afe00640 1 --2- 172.21.15.102:0/3488193886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f92b0106540 0x7f92b010cdd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009ca0 tx=0x7f92a802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.490+0000 7f92afe00640 1 -- 172.21.15.102:0/3488193886 shutdown_connections 2024-04-26T20:34:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.490+0000 7f92afe00640 1 --2- 172.21.15.102:0/3488193886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92b0105b80 0x7f92b0106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.490+0000 7f92afe00640 1 --2- 172.21.15.102:0/3488193886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f92b0106540 0x7f92b010cdd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.490+0000 7f92afe00640 1 --2- 172.21.15.102:0/3488193886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92b0104990 0x7f92b0104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.490+0000 7f92afe00640 1 -- 172.21.15.102:0/3488193886 >> 172.21.15.102:0/3488193886 conn(0x7f92b00fff00 msgr2=0x7f92b0102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.490+0000 7f92afe00640 1 -- 172.21.15.102:0/3488193886 shutdown_connections 2024-04-26T20:34:33.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.490+0000 7f92afe00640 1 -- 172.21.15.102:0/3488193886 wait complete. 2024-04-26T20:34:33.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.491+0000 7f92afe00640 1 Processor -- start 2024-04-26T20:34:33.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92afe00640 1 -- start start 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92afe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92b0104990 0x7f92b019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92afe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92b0105b80 0x7f92b019b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92afe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f92b0106540 0x7f92b01a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92afe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f92b01104f0 con 0x7f92b0105b80 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92afe00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f92b0110370 con 0x7f92b0104990 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92afe00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f92b0110670 con 0x7f92b0106540 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92ae000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92b0105b80 0x7f92b019b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92aea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92b0104990 0x7f92b019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92af400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f92b0106540 0x7f92b01a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92aea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92b0104990 0x7f92b019ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:40600/0 (socket says 172.21.15.102:40600) 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92ae000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92b0105b80 0x7f92b019b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50220/0 (socket says 172.21.15.102:50220) 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.492+0000 7f92aea00640 1 -- 172.21.15.102:0/3957724395 learned_addr learned my addr 172.21.15.102:0/3957724395 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.493+0000 7f92aea00640 1 -- 172.21.15.102:0/3957724395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f92b0106540 msgr2=0x7f92b01a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.493+0000 7f92aea00640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f92b0106540 0x7f92b01a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.493+0000 7f92aea00640 1 -- 172.21.15.102:0/3957724395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92b0105b80 msgr2=0x7f92b019b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.493+0000 7f92aea00640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92b0105b80 0x7f92b019b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.493+0000 7f92aea00640 1 -- 172.21.15.102:0/3957724395 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b01a2720 con 0x7f92b0104990 2024-04-26T20:34:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.493+0000 7f92aea00640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92b0104990 0x7f92b019ab00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f92a000e9c0 tx=0x7f92a000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.493+0000 7f92a7400640 1 -- 172.21.15.102:0/3957724395 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a000cd80 con 0x7f92b0104990 2024-04-26T20:34:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.494+0000 7f92afe00640 1 -- 172.21.15.102:0/3957724395 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b01a2a10 con 0x7f92b0104990 2024-04-26T20:34:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.494+0000 7f92a7400640 1 -- 172.21.15.102:0/3957724395 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a00035f0 con 0x7f92b0104990 2024-04-26T20:34:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.494+0000 7f92a7400640 1 -- 172.21.15.102:0/3957724395 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0010690 con 0x7f92b0104990 2024-04-26T20:34:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.494+0000 7f92afe00640 1 -- 172.21.15.102:0/3957724395 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b01a2fb0 con 0x7f92b0104990 2024-04-26T20:34:33.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.495+0000 7f92afe00640 1 -- 172.21.15.102:0/3957724395 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92b0069ac0 con 0x7f92b0104990 2024-04-26T20:34:33.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.497+0000 7f92a7400640 1 -- 172.21.15.102:0/3957724395 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f92a00037a0 con 0x7f92b0104990 2024-04-26T20:34:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.498+0000 7f92a7400640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f92940a3cc0 0x7f92940a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.498+0000 7f92a7400640 1 -- 172.21.15.102:0/3957724395 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f92a00d0c80 con 0x7f92b0104990 2024-04-26T20:34:33.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.498+0000 7f92ae000640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f92940a3cc0 0x7f92940a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.499+0000 7f92ae000640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f92940a3cc0 0x7f92940a6180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f92b019c010 tx=0x7f929c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:33.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.503+0000 7f92a7400640 1 -- 172.21.15.102:0/3957724395 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f92a009d0f0 con 0x7f92b0104990 2024-04-26T20:34:33.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.747+0000 7f92afe00640 1 -- 172.21.15.102:0/3957724395 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92b002dbb0 con 0x7f92b0104990 2024-04-26T20:34:33.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.748+0000 7f92a7400640 1 -- 172.21.15.102:0/3957724395 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11954 (secure 0 0 0) 0x7f92a009c9f0 con 0x7f92b0104990 2024-04-26T20:34:33.749 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:33.749 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:32:25.769708+0000","modified":"2024-04-26T20:32:55.699799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24593,"mds_1":24565,"mds_2":24547,"mds_3":24533,"mds_4":24526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24565,24593,24547,24533,24526]},"id":1}]} 2024-04-26T20:34:33.750 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-26T20:34:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.750+0000 7f92afe00640 1 -- 172.21.15.102:0/3957724395 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f92940a3cc0 msgr2=0x7f92940a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.750+0000 7f92afe00640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f92940a3cc0 0x7f92940a6180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f92b019c010 tx=0x7f929c005f90 comp rx=0 tx=0).stop 2024-04-26T20:34:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.751+0000 7f92afe00640 1 -- 172.21.15.102:0/3957724395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92b0104990 msgr2=0x7f92b019ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.751+0000 7f92afe00640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92b0104990 0x7f92b019ab00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f92a000e9c0 tx=0x7f92a000ee80 comp rx=0 tx=0).stop 2024-04-26T20:34:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.751+0000 7f92afe00640 1 -- 172.21.15.102:0/3957724395 shutdown_connections 2024-04-26T20:34:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.751+0000 7f92afe00640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f92940a3cc0 0x7f92940a6180 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.751+0000 7f92afe00640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92b0105b80 0x7f92b019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.751+0000 7f92afe00640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f92b0106540 0x7f92b01a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.751+0000 7f92afe00640 1 --2- 172.21.15.102:0/3957724395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f92b0104990 0x7f92b019ab00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.751+0000 7f92afe00640 1 -- 172.21.15.102:0/3957724395 >> 172.21.15.102:0/3957724395 conn(0x7f92b00fff00 msgr2=0x7f92b0108e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.751+0000 7f92afe00640 1 -- 172.21.15.102:0/3957724395 shutdown_connections 2024-04-26T20:34:33.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.751+0000 7f92afe00640 1 -- 172.21.15.102:0/3957724395 wait complete. 2024-04-26T20:34:33.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:34:33.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.964+0000 7fb192400640 1 -- 172.21.15.102:0/2783982058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18c105b80 msgr2=0x7fb18c106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.964+0000 7fb192400640 1 --2- 172.21.15.102:0/2783982058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18c105b80 0x7fb18c106000 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb174009900 tx=0x7fb17402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.964+0000 7fb192400640 1 -- 172.21.15.102:0/2783982058 shutdown_connections 2024-04-26T20:34:33.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.964+0000 7fb192400640 1 --2- 172.21.15.102:0/2783982058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18c105b80 0x7fb18c106000 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.964+0000 7fb192400640 1 --2- 172.21.15.102:0/2783982058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb18c106540 0x7fb18c10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.964+0000 7fb192400640 1 --2- 172.21.15.102:0/2783982058 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb18c104990 0x7fb18c104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.964+0000 7fb192400640 1 -- 172.21.15.102:0/2783982058 >> 172.21.15.102:0/2783982058 conn(0x7fb18c0fff00 msgr2=0x7fb18c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:33.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.964+0000 7fb192400640 1 -- 172.21.15.102:0/2783982058 shutdown_connections 2024-04-26T20:34:33.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.964+0000 7fb192400640 1 -- 172.21.15.102:0/2783982058 wait complete. 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.965+0000 7fb192400640 1 Processor -- start 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.965+0000 7fb192400640 1 -- start start 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb192400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb18c104990 0x7fb18c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb192400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb18c105b80 0x7fb18c19b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb192400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18c106540 0x7fb18c1a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb192400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c1104c0 con 0x7fb18c106540 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb192400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c110340 con 0x7fb18c104990 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb192400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c110640 con 0x7fb18c105b80 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb183e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb18c105b80 0x7fb18c19b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb191a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18c106540 0x7fb18c1a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb183e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb18c105b80 0x7fb18c19b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55726/0 (socket says 172.21.15.102:55726) 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb191000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb18c104990 0x7fb18c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.966+0000 7fb183e00640 1 -- 172.21.15.102:0/2263870613 learned_addr learned my addr 172.21.15.102:0/2263870613 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb183e00640 1 -- 172.21.15.102:0/2263870613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb18c104990 msgr2=0x7fb18c19ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb183e00640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb18c104990 0x7fb18c19ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb183e00640 1 -- 172.21.15.102:0/2263870613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18c106540 msgr2=0x7fb18c1a2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb183e00640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18c106540 0x7fb18c1a2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb183e00640 1 -- 172.21.15.102:0/2263870613 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18c1a2790 con 0x7fb18c105b80 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb191000640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb18c104990 0x7fb18c19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb183e00640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb18c105b80 0x7fb18c19b0b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb174005940 tx=0x7fb1740042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb18a000640 1 -- 172.21.15.102:0/2263870613 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17403f030 con 0x7fb18c105b80 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb192400640 1 -- 172.21.15.102:0/2263870613 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18c1a2a20 con 0x7fb18c105b80 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb18a000640 1 -- 172.21.15.102:0/2263870613 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb174003cb0 con 0x7fb18c105b80 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb18a000640 1 -- 172.21.15.102:0/2263870613 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb174035070 con 0x7fb18c105b80 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.967+0000 7fb192400640 1 -- 172.21.15.102:0/2263870613 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18c1a2f60 con 0x7fb18c105b80 2024-04-26T20:34:33.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.968+0000 7fb192400640 1 -- 172.21.15.102:0/2263870613 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb18c10bd30 con 0x7fb18c105b80 2024-04-26T20:34:33.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.971+0000 7fb18a000640 1 -- 172.21.15.102:0/2263870613 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fb174042030 con 0x7fb18c105b80 2024-04-26T20:34:33.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.972+0000 7fb18a000640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb1700a3c40 0x7fb1700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:33.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.972+0000 7fb18a000640 1 -- 172.21.15.102:0/2263870613 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fb1740f11b0 con 0x7fb18c105b80 2024-04-26T20:34:33.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.972+0000 7fb191000640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb1700a3c40 0x7fb1700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:33.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.973+0000 7fb191000640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb1700a3c40 0x7fb1700a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb18c1059e0 tx=0x7fb17c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:33.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:33.975+0000 7fb18a000640 1 -- 172.21.15.102:0/2263870613 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb174003e60 con 0x7fb18c105b80 2024-04-26T20:34:34.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:33 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1466409124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:34:34.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:33 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3253928532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:34:34.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:33 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1466409124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:34:34.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:33 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3253928532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:34:34.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:33 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1466409124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:34:34.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:33 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3253928532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:34:34.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.165+0000 7fb192400640 1 -- 172.21.15.102:0/2263870613 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb18c02dcd0 con 0x7fb18c105b80 2024-04-26T20:34:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.167+0000 7fb18a000640 1 -- 172.21.15.102:0/2263870613 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24833 (secure 0 0 0) 0x7fb1740bcfd0 con 0x7fb18c105b80 2024-04-26T20:34:34.168 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:34.168 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:34:30.697401+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.169+0000 7fb192400640 1 -- 172.21.15.102:0/2263870613 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb1700a3c40 msgr2=0x7fb1700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.169+0000 7fb192400640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb1700a3c40 0x7fb1700a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb18c1059e0 tx=0x7fb17c005f90 comp rx=0 tx=0).stop 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.169+0000 7fb192400640 1 -- 172.21.15.102:0/2263870613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb18c105b80 msgr2=0x7fb18c19b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.169+0000 7fb192400640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb18c105b80 0x7fb18c19b0b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb174005940 tx=0x7fb1740042f0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.170+0000 7fb192400640 1 -- 172.21.15.102:0/2263870613 shutdown_connections 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.170+0000 7fb192400640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb1700a3c40 0x7fb1700a6100 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.170+0000 7fb192400640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18c106540 0x7fb18c1a2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.170+0000 7fb192400640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb18c105b80 0x7fb18c19b0b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.170+0000 7fb192400640 1 --2- 172.21.15.102:0/2263870613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb18c104990 0x7fb18c19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.170+0000 7fb192400640 1 -- 172.21.15.102:0/2263870613 >> 172.21.15.102:0/2263870613 conn(0x7fb18c0fff00 msgr2=0x7fb18c108f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.170+0000 7fb192400640 1 -- 172.21.15.102:0/2263870613 shutdown_connections 2024-04-26T20:34:34.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.170+0000 7fb192400640 1 -- 172.21.15.102:0/2263870613 wait complete. 2024-04-26T20:34:34.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:34:34.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.378+0000 7fb528c00640 1 -- 172.21.15.102:0/3264674983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb524102770 msgr2=0x7fb524102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.378+0000 7fb528c00640 1 --2- 172.21.15.102:0/3264674983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb524102770 0x7fb524102b90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb51c009ca0 tx=0x7fb51c02a8c0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.378+0000 7fb528c00640 1 -- 172.21.15.102:0/3264674983 shutdown_connections 2024-04-26T20:34:34.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.378+0000 7fb528c00640 1 --2- 172.21.15.102:0/3264674983 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb524104320 0x7fb52410abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.378+0000 7fb528c00640 1 --2- 172.21.15.102:0/3264674983 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb524103960 0x7fb524103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.378+0000 7fb528c00640 1 --2- 172.21.15.102:0/3264674983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb524102770 0x7fb524102b90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.378+0000 7fb528c00640 1 -- 172.21.15.102:0/3264674983 >> 172.21.15.102:0/3264674983 conn(0x7fb5240fdcc0 msgr2=0x7fb524100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:34.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.378+0000 7fb528c00640 1 -- 172.21.15.102:0/3264674983 shutdown_connections 2024-04-26T20:34:34.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.378+0000 7fb528c00640 1 -- 172.21.15.102:0/3264674983 wait complete. 2024-04-26T20:34:34.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.379+0000 7fb528c00640 1 Processor -- start 2024-04-26T20:34:34.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb528c00640 1 -- start start 2024-04-26T20:34:34.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb528c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb524102770 0x7fb52419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb528c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb524103960 0x7fb52419b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb528c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb524104320 0x7fb5241a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb528c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb52410e2a0 con 0x7fb524103960 2024-04-26T20:34:34.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb528c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb52410e120 con 0x7fb524102770 2024-04-26T20:34:34.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb528c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb52410e420 con 0x7fb524104320 2024-04-26T20:34:34.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb523400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb524102770 0x7fb52419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb522a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb524103960 0x7fb52419b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb523400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb524102770 0x7fb52419ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:40620/0 (socket says 172.21.15.102:40620) 2024-04-26T20:34:34.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.380+0000 7fb523400640 1 -- 172.21.15.102:0/4038491137 learned_addr learned my addr 172.21.15.102:0/4038491137 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:34.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.381+0000 7fb523e00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb524104320 0x7fb5241a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.381+0000 7fb523400640 1 -- 172.21.15.102:0/4038491137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb524104320 msgr2=0x7fb5241a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.381+0000 7fb523400640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb524104320 0x7fb5241a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.381+0000 7fb523400640 1 -- 172.21.15.102:0/4038491137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb524103960 msgr2=0x7fb52419b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.381+0000 7fb523400640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb524103960 0x7fb52419b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.381+0000 7fb523400640 1 -- 172.21.15.102:0/4038491137 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5241a27a0 con 0x7fb524102770 2024-04-26T20:34:34.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.381+0000 7fb522a00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb524103960 0x7fb52419b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:34.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.381+0000 7fb523400640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb524102770 0x7fb52419ab80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb51c009430 tx=0x7fb51c0316e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:34.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.381+0000 7fb51be00640 1 -- 172.21.15.102:0/4038491137 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c003b70 con 0x7fb524102770 2024-04-26T20:34:34.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.382+0000 7fb51be00640 1 -- 172.21.15.102:0/4038491137 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb51c003d10 con 0x7fb524102770 2024-04-26T20:34:34.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.382+0000 7fb528c00640 1 -- 172.21.15.102:0/4038491137 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5241a2a30 con 0x7fb524102770 2024-04-26T20:34:34.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.382+0000 7fb51be00640 1 -- 172.21.15.102:0/4038491137 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c02fb60 con 0x7fb524102770 2024-04-26T20:34:34.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.382+0000 7fb528c00640 1 -- 172.21.15.102:0/4038491137 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5241a2f80 con 0x7fb524102770 2024-04-26T20:34:34.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.383+0000 7fb528c00640 1 -- 172.21.15.102:0/4038491137 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5241a31b0 con 0x7fb524102770 2024-04-26T20:34:34.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.385+0000 7fb51be00640 1 -- 172.21.15.102:0/4038491137 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fb51c02fd00 con 0x7fb524102770 2024-04-26T20:34:34.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.387+0000 7fb51be00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5080a3c40 0x7fb5080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.387+0000 7fb51be00640 1 -- 172.21.15.102:0/4038491137 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fb51c0f1e10 con 0x7fb524102770 2024-04-26T20:34:34.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.387+0000 7fb522a00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5080a3c40 0x7fb5080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.388+0000 7fb522a00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5080a3c40 0x7fb5080a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb52419c090 tx=0x7fb50c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:34.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.388+0000 7fb51be00640 1 -- 172.21.15.102:0/4038491137 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb51c0be390 con 0x7fb524102770 2024-04-26T20:34:34.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.636+0000 7fb528c00640 1 -- 172.21.15.102:0/4038491137 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5241a34a0 con 0x7fb524102770 2024-04-26T20:34:34.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.637+0000 7fb51be00640 1 -- 172.21.15.102:0/4038491137 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11954 (secure 0 0 0) 0x7fb51c0bdc90 con 0x7fb524102770 2024-04-26T20:34:34.639 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:34.639 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:32:25.769708+0000","modified":"2024-04-26T20:32:55.699799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24593,"mds_1":24565,"mds_2":24547,"mds_3":24533,"mds_4":24526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24565,24593,24547,24533,24526]},"id":1}]} 2024-04-26T20:34:34.639 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-26T20:34:34.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 -- 172.21.15.102:0/4038491137 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5080a3c40 msgr2=0x7fb5080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5080a3c40 0x7fb5080a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb52419c090 tx=0x7fb50c005f90 comp rx=0 tx=0).stop 2024-04-26T20:34:34.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 -- 172.21.15.102:0/4038491137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb524102770 msgr2=0x7fb52419ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb524102770 0x7fb52419ab80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb51c009430 tx=0x7fb51c0316e0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 -- 172.21.15.102:0/4038491137 shutdown_connections 2024-04-26T20:34:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fb5080a3c40 0x7fb5080a6100 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb524103960 0x7fb52419b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb524104320 0x7fb5241a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 --2- 172.21.15.102:0/4038491137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb524102770 0x7fb52419ab80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 -- 172.21.15.102:0/4038491137 >> 172.21.15.102:0/4038491137 conn(0x7fb5240fdcc0 msgr2=0x7fb524074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:34.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 -- 172.21.15.102:0/4038491137 shutdown_connections 2024-04-26T20:34:34.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.640+0000 7fb528c00640 1 -- 172.21.15.102:0/4038491137 wait complete. 2024-04-26T20:34:34.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T20:32:25.769708+0000', 'modified': '2024-04-26T20:32:55.699799+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24593, 'mds_1': 24565, 'mds_2': 24547, 'mds_3': 24533, 'mds_4': 24526}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24448': {'gid': 24448, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6833/4251888549', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6832', 'nonce': 4251888549}, {'type': 'v1', 'addr': '172.21.15.130:6833', 'nonce': 4251888549}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6833/1543441778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6832', 'nonce': 1543441778}, {'type': 'v1', 'addr': '172.21.15.112:6833', 'nonce': 1543441778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'd', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.102:6837/3283929332', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 3283929332}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 3283929332}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6835/2010209065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6834', 'nonce': 2010209065}, {'type': 'v1', 'addr': '172.21.15.112:6835', 'nonce': 2010209065}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24514': {'gid': 24514, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6835/829567775', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 829567775}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 829567775}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24526': {'gid': 24526, 'name': 'c', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.130:6837/2107305806', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 2107305806}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 2107305806}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.112:6837/82964069', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6836', 'nonce': 82964069}, {'type': 'v1', 'addr': '172.21.15.112:6837', 'nonce': 82964069}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.130:6839/1700008514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 1700008514}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 1700008514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24565': {'gid': 24565, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.102:6839/3473487681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6838', 'nonce': 3473487681}, {'type': 'v1', 'addr': '172.21.15.102:6839', 'nonce': 3473487681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24593': {'gid': 24593, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.112:6839/2318597180', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 2318597180}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 2318597180}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24593, 'qdb_cluster': [24565, 24593, 24547, 24533, 24526]} 2024-04-26T20:34:34.662 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-26T20:34:34.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T20:34:34.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.864+0000 7fd7afe00640 1 -- 172.21.15.102:0/3972137086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7b0102770 msgr2=0x7fd7b0102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.864+0000 7fd7afe00640 1 --2- 172.21.15.102:0/3972137086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7b0102770 0x7fd7b0102b90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0009900 tx=0x7fd7a002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.864+0000 7fd7afe00640 1 -- 172.21.15.102:0/3972137086 shutdown_connections 2024-04-26T20:34:34.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.864+0000 7fd7afe00640 1 --2- 172.21.15.102:0/3972137086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7b0104320 0x7fd7b010abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.864+0000 7fd7afe00640 1 --2- 172.21.15.102:0/3972137086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b0103960 0x7fd7b0103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.864+0000 7fd7afe00640 1 --2- 172.21.15.102:0/3972137086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7b0102770 0x7fd7b0102b90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.864+0000 7fd7afe00640 1 -- 172.21.15.102:0/3972137086 >> 172.21.15.102:0/3972137086 conn(0x7fd7b00fdce0 msgr2=0x7fd7b0100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:34.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.864+0000 7fd7afe00640 1 -- 172.21.15.102:0/3972137086 shutdown_connections 2024-04-26T20:34:34.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.865+0000 7fd7afe00640 1 -- 172.21.15.102:0/3972137086 wait complete. 2024-04-26T20:34:34.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.865+0000 7fd7afe00640 1 Processor -- start 2024-04-26T20:34:34.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7afe00640 1 -- start start 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7afe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7b0102770 0x7fd7b019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7afe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7b0103960 0x7fd7b019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7afe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b0104320 0x7fd7b01a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7afe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b010e270 con 0x7fd7b0102770 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7afe00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b010e0f0 con 0x7fd7b0103960 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7afe00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b010e3f0 con 0x7fd7b0104320 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7aea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7b0102770 0x7fd7b019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7aea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7b0102770 0x7fd7b019abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37848/0 (socket says 172.21.15.102:37848) 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7aea00640 1 -- 172.21.15.102:0/3393421998 learned_addr learned my addr 172.21.15.102:0/3393421998 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7ae000640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7b0103960 0x7fd7b019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.866+0000 7fd7af400640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b0104320 0x7fd7b01a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.867+0000 7fd7aea00640 1 -- 172.21.15.102:0/3393421998 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b0104320 msgr2=0x7fd7b01a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.867+0000 7fd7aea00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b0104320 0x7fd7b01a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.867+0000 7fd7aea00640 1 -- 172.21.15.102:0/3393421998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7b0103960 msgr2=0x7fd7b019b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.867+0000 7fd7aea00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7b0103960 0x7fd7b019b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.867+0000 7fd7aea00640 1 -- 172.21.15.102:0/3393421998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b01a27f0 con 0x7fd7b0102770 2024-04-26T20:34:34.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.867+0000 7fd7aea00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7b0102770 0x7fd7b019abd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0006010 tx=0x7fd7a00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:34.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.867+0000 7fd7a7400640 1 -- 172.21.15.102:0/3393421998 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a003f030 con 0x7fd7b0102770 2024-04-26T20:34:34.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.867+0000 7fd7afe00640 1 -- 172.21.15.102:0/3393421998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b01a2a80 con 0x7fd7b0102770 2024-04-26T20:34:34.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.867+0000 7fd7a7400640 1 -- 172.21.15.102:0/3393421998 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7a0031910 con 0x7fd7b0102770 2024-04-26T20:34:34.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.868+0000 7fd7a7400640 1 -- 172.21.15.102:0/3393421998 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0049060 con 0x7fd7b0102770 2024-04-26T20:34:34.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.868+0000 7fd7afe00640 1 -- 172.21.15.102:0/3393421998 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b01a2fc0 con 0x7fd7b0102770 2024-04-26T20:34:34.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.871+0000 7fd7a7400640 1 -- 172.21.15.102:0/3393421998 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fd7a0042030 con 0x7fd7b0102770 2024-04-26T20:34:34.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.871+0000 7fd7b85f7740 1 Processor -- start 2024-04-26T20:34:34.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.871+0000 7fd7b85f7740 1 -- start start 2024-04-26T20:34:34.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.871+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df5c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.871+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 0x55c06df617b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.871+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df61e20 0x55c06df642b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c06dec0600 con 0x55c06df5ccb0 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c06de5d100 con 0x55c06df61e20 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c06debc5c0 con 0x55c06df5c2c0 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7af400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df5c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7af400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df5c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55560/0 (socket says 172.21.15.102:55560) 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7af400640 1 -- 172.21.15.102:0/633180586 learned_addr learned my addr 172.21.15.102:0/633180586 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7aea00640 1 --2- 172.21.15.102:0/633180586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 0x55c06df617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7ae000640 1 --2- 172.21.15.102:0/633180586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df61e20 0x55c06df642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7af400640 1 -- 172.21.15.102:0/633180586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df61e20 msgr2=0x55c06df642b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7af400640 1 --2- 172.21.15.102:0/633180586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df61e20 0x55c06df642b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7af400640 1 -- 172.21.15.102:0/633180586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 msgr2=0x55c06df617b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:34.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7af400640 1 --2- 172.21.15.102:0/633180586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 0x55c06df617b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.872+0000 7fd7af400640 1 -- 172.21.15.102:0/633180586 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c06dbd22d0 con 0x55c06df5c2c0 2024-04-26T20:34:34.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.873+0000 7fd7af400640 1 --2- 172.21.15.102:0/633180586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df5c6e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8011440 tx=0x7fd7a8011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d001ebe09c45763f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:34.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.873+0000 7fd7a7400640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd7940a3c40 0x7fd7940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.873+0000 7fd7a7400640 1 -- 172.21.15.102:0/3393421998 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fd7a00f1140 con 0x7fd7b0102770 2024-04-26T20:34:34.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.873+0000 7fd79be00640 1 -- 172.21.15.102:0/633180586 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8003c30 con 0x55c06df5c2c0 2024-04-26T20:34:34.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.873+0000 7fd79be00640 1 -- 172.21.15.102:0/633180586 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7a8003dd0 con 0x55c06df5c2c0 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.873+0000 7fd79be00640 1 -- 172.21.15.102:0/633180586 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a800fae0 con 0x55c06df5c2c0 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.873+0000 7fd7aea00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd7940a3c40 0x7fd7940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7b85f7740 1 -- 172.21.15.102:0/633180586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 msgr2=0x55c06df5c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/633180586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df5c6e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8011440 tx=0x7fd7a8011900 comp rx=0 tx=0).stop 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7aea00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd7940a3c40 0x7fd7940a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55c06df5d0b0 tx=0x7fd7a00f4bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7b85f7740 1 -- 172.21.15.102:0/633180586 shutdown_connections 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/633180586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 0x55c06df617b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/633180586 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df61e20 0x55c06df642b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/633180586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df5c6e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7b85f7740 1 -- 172.21.15.102:0/633180586 >> 172.21.15.102:0/633180586 conn(0x55c06def0d10 msgr2=0x55c06df57d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7b85f7740 1 -- 172.21.15.102:0/633180586 shutdown_connections 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7b85f7740 1 -- 172.21.15.102:0/633180586 wait complete. 2024-04-26T20:34:34.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.874+0000 7fd7b85f7740 1 -- start start 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.875+0000 7fd7b85f7740 1 Processor -- start 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.875+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5c2c0 0x55c06df66740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.875+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 0x55c06df66c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.875+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 0x55c06df6dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.875+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c06df64dd0 con 0x55c06df5c2c0 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.875+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c06de47f70 con 0x55c06df5ccb0 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.875+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c06df64f50 con 0x55c06df61e20 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.875+0000 7fd7b85f7740 10 client.0 _ll_register_callbacks cb 0x55c06deb6480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.875+0000 7fd7af400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5c2c0 0x55c06df66740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7af400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5c2c0 0x55c06df66740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37874/0 (socket says 172.21.15.102:37874) 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7ae000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 0x55c06df66c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7af400640 1 -- 172.21.15.102:0/388607149 learned_addr learned my addr 172.21.15.102:0/388607149 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7af400640 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 0x55c06df6dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7af400640 1 -- 172.21.15.102:0/388607149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 msgr2=0x55c06df6dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7af400640 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 0x55c06df6dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7af400640 1 -- 172.21.15.102:0/388607149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 msgr2=0x55c06df66c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7af400640 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 0x55c06df66c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7af400640 1 -- 172.21.15.102:0/388607149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c06dc9c6b0 con 0x55c06df5c2c0 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7ae000640 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 0x55c06df66c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd7af400640 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5c2c0 0x55c06df66740 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009c30 tx=0x7fd7a8009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334e49846440c5ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd798c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd798c00640 1 -- 172.21.15.102:0/388607149 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a800fab0 con 0x55c06df5c2c0 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.876+0000 7fd798c00640 1 -- 172.21.15.102:0/388607149 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fd7a800fc50 con 0x55c06df5c2c0 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd7b85f7740 1 -- 172.21.15.102:0/388607149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7a8003dd0 con 0x55c06df5c2c0 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd798c00640 1 -- 172.21.15.102:0/388607149 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd7a8036050 con 0x55c06df5c2c0 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd7b85f7740 10 client.15042 fetch_fsmap learned FSMap version 27 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd7b85f7740 1 -- 172.21.15.102:0/388607149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c06df70ca0 con 0x55c06df5c2c0 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd798c00640 1 -- 172.21.15.102:0/388607149 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7a8003dd0 con 0x55c06df5c2c0 2024-04-26T20:34:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd798c00640 1 -- 172.21.15.102:0/388607149 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd7a8025040 con 0x55c06df5c2c0 2024-04-26T20:34:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd7b85f7740 10 client.15042 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:34:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd7b85f7740 10 client.15042 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T20:34:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd7b85f7740 20 client.15042 populate_metadata read hostname 'smithi102' 2024-04-26T20:34:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x55c06df730d0 0x55c06df74540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd7b85f7740 4 client.15042 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:34:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.877+0000 7fd7b85f7740 1 -- 172.21.15.102:0/388607149 --> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c06dece210 con 0x55c06df730d0 2024-04-26T20:34:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.878+0000 7fd7ae000640 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x55c06df730d0 0x55c06df74540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.878+0000 7fd7ae000640 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x55c06df730d0 0x55c06df74540 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dec9ec8e3a6ec1b5 server_cookie=46605f7128b3b873 in_seq=0 out_seq=0 2024-04-26T20:34:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.878+0000 7fd798c00640 10 client.15042 ms_handle_connect on v2:172.21.15.130:6836/2107305806 2024-04-26T20:34:34.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.879+0000 7fd798c00640 1 -- 172.21.15.102:0/388607149 <== mds.4 v2:172.21.15.130:6836/2107305806 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c06dece210 con 0x55c06df730d0 2024-04-26T20:34:34.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.879+0000 7fd798c00640 10 client.15042 handle_command_reply: tid=0 2024-04-26T20:34:34.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.879+0000 7fd7b85f7740 1 client.15042 shutdown 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.879+0000 7fd7b85f7740 1 -- 172.21.15.102:0/388607149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5c2c0 msgr2=0x55c06df66740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.879+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5c2c0 0x55c06df66740 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009c30 tx=0x7fd7a8009fb0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.879+0000 7fd7b85f7740 1 -- 172.21.15.102:0/388607149 shutdown_connections 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.879+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x55c06df730d0 0x55c06df74540 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.879+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 0x55c06df6dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.879+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 0x55c06df66c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.880+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/388607149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5c2c0 0x55c06df66740 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.880+0000 7fd7b85f7740 1 -- 172.21.15.102:0/388607149 >> 172.21.15.102:0/388607149 conn(0x55c06def0d10 msgr2=0x55c06df56f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.880+0000 7fd798c00640 0 client.15042 ms_handle_reset on v2:172.21.15.130:6836/2107305806 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.880+0000 7fd7b85f7740 1 -- 172.21.15.102:0/388607149 shutdown_connections 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.880+0000 7fd7b85f7740 1 -- 172.21.15.102:0/388607149 wait complete. 2024-04-26T20:34:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.880+0000 7fd7b85f7740 20 client.15042 trim_cache size 0 max 16384 2024-04-26T20:34:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7b85f7740 1 Processor -- start 2024-04-26T20:34:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7b85f7740 1 -- start start 2024-04-26T20:34:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df66a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 0x55c06df6dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df61e20 0x55c06df7b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c06df72880 con 0x55c06df61e20 2024-04-26T20:34:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c06df72f50 con 0x55c06df5ccb0 2024-04-26T20:34:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c06df73620 con 0x55c06df5c2c0 2024-04-26T20:34:34.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7af400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df61e20 0x55c06df7b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7ae000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 0x55c06df6dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7af400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df66a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7af400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df66a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55572/0 (socket says 172.21.15.102:55572) 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.899+0000 7fd7af400640 1 -- 172.21.15.102:0/3159775367 learned_addr learned my addr 172.21.15.102:0/3159775367 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.900+0000 7fd7ae000640 1 -- 172.21.15.102:0/3159775367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 msgr2=0x55c06df66a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.900+0000 7fd7ae000640 1 --2- 172.21.15.102:0/3159775367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df66a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.900+0000 7fd7ae000640 1 -- 172.21.15.102:0/3159775367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df61e20 msgr2=0x55c06df7b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.900+0000 7fd7ae000640 1 --2- 172.21.15.102:0/3159775367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df61e20 0x55c06df7b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.900+0000 7fd7ae000640 1 -- 172.21.15.102:0/3159775367 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c06df589f0 con 0x55c06df5ccb0 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.900+0000 7fd7ae000640 1 --2- 172.21.15.102:0/3159775367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 0x55c06df6dc80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55c06df74c80 tx=0x7fd79c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df738838b61e012c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.900+0000 7fd79b400640 1 -- 172.21.15.102:0/3159775367 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c0122d0 con 0x55c06df5ccb0 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.900+0000 7fd79b400640 1 -- 172.21.15.102:0/3159775367 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd79c026070 con 0x55c06df5ccb0 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.900+0000 7fd79b400640 1 -- 172.21.15.102:0/3159775367 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c005020 con 0x55c06df5ccb0 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.901+0000 7fd7b85f7740 1 -- 172.21.15.102:0/3159775367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 msgr2=0x55c06df6dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.901+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/3159775367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 0x55c06df6dc80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55c06df74c80 tx=0x7fd79c005f90 comp rx=0 tx=0).stop 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.901+0000 7fd7b85f7740 1 -- 172.21.15.102:0/3159775367 shutdown_connections 2024-04-26T20:34:34.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.901+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/3159775367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df61e20 0x55c06df7b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.901+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/3159775367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5ccb0 0x55c06df6dc80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.901+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/3159775367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df5c2c0 0x55c06df66a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.901+0000 7fd7b85f7740 1 -- 172.21.15.102:0/3159775367 >> 172.21.15.102:0/3159775367 conn(0x55c06def0d10 msgr2=0x55c06df66350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:34.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.901+0000 7fd7b85f7740 1 -- 172.21.15.102:0/3159775367 shutdown_connections 2024-04-26T20:34:34.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.901+0000 7fd7b85f7740 1 -- 172.21.15.102:0/3159775367 wait complete. 2024-04-26T20:34:34.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.901+0000 7fd7b85f7740 1 -- start start 2024-04-26T20:34:34.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7b85f7740 1 Processor -- start 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5c2c0 0x55c06df69720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 0x55c06df69c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7b85f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 0x55c06df6cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c06df743c0 con 0x55c06df5ccb0 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c06df73cf0 con 0x55c06df5c2c0 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7b85f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c06df74a90 con 0x55c06df61e20 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7b85f7740 10 client.0 _ll_register_callbacks cb 0x7fd7a00f4ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7ae000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 0x55c06df69c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7ae000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 0x55c06df69c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37896/0 (socket says 172.21.15.102:37896) 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7ae000640 1 -- 172.21.15.102:0/4202039476 learned_addr learned my addr 172.21.15.102:0/4202039476 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.902+0000 7fd7af400640 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5c2c0 0x55c06df69720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7af400640 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 0x55c06df6cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7ae000640 1 -- 172.21.15.102:0/4202039476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 msgr2=0x55c06df6cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7ae000640 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 0x55c06df6cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7ae000640 1 -- 172.21.15.102:0/4202039476 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5c2c0 msgr2=0x55c06df69720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7ae000640 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5c2c0 0x55c06df69720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7ae000640 1 -- 172.21.15.102:0/4202039476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c06df6d270 con 0x55c06df5ccb0 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7af400640 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 0x55c06df6cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7ae000640 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 0x55c06df69c60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55c06df6ed90 tx=0x7fd79c0126d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea8799312eb06d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd798c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd798c00640 1 -- 172.21.15.102:0/4202039476 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c02e070 con 0x55c06df5ccb0 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd798c00640 1 -- 172.21.15.102:0/4202039476 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fd79c0122d0 con 0x55c06df5ccb0 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7b85f7740 1 -- 172.21.15.102:0/4202039476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd79c026070 con 0x55c06df5ccb0 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd798c00640 1 -- 172.21.15.102:0/4202039476 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd79c038050 con 0x55c06df5ccb0 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7b85f7740 10 client.15048 fetch_fsmap learned FSMap version 27 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.903+0000 7fd7b85f7740 1 -- 172.21.15.102:0/4202039476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c06dfe1c20 con 0x55c06df5ccb0 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.904+0000 7fd798c00640 1 -- 172.21.15.102:0/4202039476 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd79c026070 con 0x55c06df5ccb0 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.904+0000 7fd798c00640 1 -- 172.21.15.102:0/4202039476 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd79c0046c0 con 0x55c06df5ccb0 2024-04-26T20:34:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.904+0000 7fd7b85f7740 10 client.15048 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.904+0000 7fd7b85f7740 10 client.15048 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.904+0000 7fd7b85f7740 20 client.15048 populate_metadata read hostname 'smithi102' 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.904+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x55c06dfe6500 0x55c06dfe5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.904+0000 7fd7b85f7740 4 client.15048 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.904+0000 7fd7b85f7740 1 -- 172.21.15.102:0/4202039476 --> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] -- command(tid 0: {"prefix": "status"}) -- 0x55c06dec8080 con 0x55c06dfe6500 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.904+0000 7fd7af400640 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x55c06dfe6500 0x55c06dfe5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd7af400640 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x55c06dfe6500 0x55c06dfe5210 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8489dd3e5c13016a server_cookie=b83470dd94ecc2aa in_seq=0 out_seq=0 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd798c00640 10 client.15048 ms_handle_connect on v2:172.21.15.130:6836/2107305806 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd798c00640 1 -- 172.21.15.102:0/4202039476 <== mds.4 v2:172.21.15.130:6836/2107305806 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55c06dec8080 con 0x55c06dfe6500 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd798c00640 10 client.15048 handle_command_reply: tid=0 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd7b85f7740 1 client.15048 shutdown 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd7b85f7740 1 -- 172.21.15.102:0/4202039476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 msgr2=0x55c06df69c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 0x55c06df69c60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55c06df6ed90 tx=0x7fd79c0126d0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd7b85f7740 1 -- 172.21.15.102:0/4202039476 shutdown_connections 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x55c06dfe6500 0x55c06dfe5210 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c06df5ccb0 0x55c06df69c60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c06df61e20 0x55c06df6cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd798c00640 0 client.15048 ms_handle_reset on v2:172.21.15.130:6836/2107305806 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd7b85f7740 1 --2- 172.21.15.102:0/4202039476 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c06df5c2c0 0x55c06df69720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.905+0000 7fd7b85f7740 1 -- 172.21.15.102:0/4202039476 >> 172.21.15.102:0/4202039476 conn(0x55c06def0d10 msgr2=0x55c06df5ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7b85f7740 1 -- 172.21.15.102:0/4202039476 shutdown_connections 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7b85f7740 1 -- 172.21.15.102:0/4202039476 wait complete. 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7b85f7740 20 client.15048 trim_cache size 0 max 16384 2024-04-26T20:34:34.911 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 4, 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24526, 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 100.231096059, 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 27, 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 151.99731391 2024-04-26T20:34:34.912 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:34:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7afe00640 1 -- 172.21.15.102:0/3393421998 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd7940a3c40 msgr2=0x7fd7940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7afe00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd7940a3c40 0x7fd7940a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55c06df5d0b0 tx=0x7fd7a00f4bf0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7afe00640 1 -- 172.21.15.102:0/3393421998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7b0102770 msgr2=0x7fd7b019abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7afe00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7b0102770 0x7fd7b019abd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0006010 tx=0x7fd7a00042f0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7afe00640 1 -- 172.21.15.102:0/3393421998 shutdown_connections 2024-04-26T20:34:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7afe00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd7940a3c40 0x7fd7940a6100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7afe00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b0104320 0x7fd7b01a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7afe00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7b0103960 0x7fd7b019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7afe00640 1 --2- 172.21.15.102:0/3393421998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7b0102770 0x7fd7b019abd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.906+0000 7fd7afe00640 1 -- 172.21.15.102:0/3393421998 >> 172.21.15.102:0/3393421998 conn(0x7fd7b00fdce0 msgr2=0x7fd7b0109180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.907+0000 7fd7afe00640 1 -- 172.21.15.102:0/3393421998 shutdown_connections 2024-04-26T20:34:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:34.907+0000 7fd7afe00640 1 -- 172.21.15.102:0/3393421998 wait complete. 2024-04-26T20:34:34.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:34 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3957724395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:34:34.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:34 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2263870613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:34:34.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:34 smithi102 ceph-mon[28094]: pgmap v386: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s, 1 objects/s recovering 2024-04-26T20:34:34.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:34 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/4038491137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:34:34.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-26T20:34:35.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:34 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3957724395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:34:35.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:34 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2263870613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:34:35.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:34 smithi112 ceph-mon[32769]: pgmap v386: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s, 1 objects/s recovering 2024-04-26T20:34:35.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:34 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/4038491137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:34:35.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.115+0000 7f5417e00640 1 -- 172.21.15.102:0/831411101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5418105b80 msgr2=0x7f5418106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.115+0000 7f5417e00640 1 --2- 172.21.15.102:0/831411101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5418105b80 0x7f5418106000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5400009900 tx=0x7f540002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.115+0000 7f5417e00640 1 -- 172.21.15.102:0/831411101 shutdown_connections 2024-04-26T20:34:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.115+0000 7f5417e00640 1 --2- 172.21.15.102:0/831411101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5418106540 0x7f541810cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.115+0000 7f5417e00640 1 --2- 172.21.15.102:0/831411101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5418105b80 0x7f5418106000 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.115+0000 7f5417e00640 1 --2- 172.21.15.102:0/831411101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418104990 0x7f5418104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.115+0000 7f5417e00640 1 -- 172.21.15.102:0/831411101 >> 172.21.15.102:0/831411101 conn(0x7f54180fff00 msgr2=0x7f5418102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.115+0000 7f5417e00640 1 -- 172.21.15.102:0/831411101 shutdown_connections 2024-04-26T20:34:35.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.115+0000 7f5417e00640 1 -- 172.21.15.102:0/831411101 wait complete. 2024-04-26T20:34:35.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.116+0000 7f5417e00640 1 Processor -- start 2024-04-26T20:34:35.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.116+0000 7f5417e00640 1 -- start start 2024-04-26T20:34:35.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5417e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418104990 0x7f541819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5417e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5418105b80 0x7f541819b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5417e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5418106540 0x7f54181a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5417e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f54181104f0 con 0x7f5418104990 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5417e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5418110370 con 0x7f5418106540 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5417e00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5418110670 con 0x7f5418105b80 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5416a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418104990 0x7f541819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5416a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418104990 0x7f541819aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37902/0 (socket says 172.21.15.102:37902) 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5416a00640 1 -- 172.21.15.102:0/3800495906 learned_addr learned my addr 172.21.15.102:0/3800495906 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5417400640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5418106540 0x7f54181a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f540fe00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5418105b80 0x7f541819b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5416a00640 1 -- 172.21.15.102:0/3800495906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5418105b80 msgr2=0x7f541819b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5416a00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5418105b80 0x7f541819b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5416a00640 1 -- 172.21.15.102:0/3800495906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5418106540 msgr2=0x7f54181a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5416a00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5418106540 0x7f54181a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f5416a00640 1 -- 172.21.15.102:0/3800495906 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54181a27c0 con 0x7f5418104990 2024-04-26T20:34:35.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.117+0000 7f540fe00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5418105b80 0x7f541819b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:34:35.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.118+0000 7f5416a00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418104990 0x7f541819aba0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f540800e9c0 tx=0x7f540800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.118+0000 7f540f400640 1 -- 172.21.15.102:0/3800495906 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540800cd80 con 0x7f5418104990 2024-04-26T20:34:35.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.118+0000 7f5417e00640 1 -- 172.21.15.102:0/3800495906 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54181a2ab0 con 0x7f5418104990 2024-04-26T20:34:35.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.118+0000 7f540f400640 1 -- 172.21.15.102:0/3800495906 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54080035f0 con 0x7f5418104990 2024-04-26T20:34:35.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.118+0000 7f540f400640 1 -- 172.21.15.102:0/3800495906 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408010690 con 0x7f5418104990 2024-04-26T20:34:35.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.118+0000 7f5417e00640 1 -- 172.21.15.102:0/3800495906 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54181a2fd0 con 0x7f5418104990 2024-04-26T20:34:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:34 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3957724395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:34:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:34 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2263870613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:34:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:34 smithi130 ceph-mon[32704]: pgmap v386: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s, 1 objects/s recovering 2024-04-26T20:34:35.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:34 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/4038491137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:34:35.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.121+0000 7f540f400640 1 -- 172.21.15.102:0/3800495906 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f54080037a0 con 0x7f5418104990 2024-04-26T20:34:35.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f54205fe740 1 Processor -- start 2024-04-26T20:34:35.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f54205fe740 1 -- start start 2024-04-26T20:34:35.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f54205fe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6566e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f54205fe740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f65b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f54205fe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f65e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f54205fe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f5ba600 con 0x55cc3f65be20 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f54205fe740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f557100 con 0x55cc3f656cb0 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f54205fe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f5b65c0 con 0x55cc3f6562c0 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f5416a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f65b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f5417400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f5417400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6566e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55616/0 (socket says 172.21.15.102:55616) 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f5416a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f65b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:45670/0 (socket says 172.21.15.102:45670) 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f5417400640 1 -- 172.21.15.102:0/3847376734 learned_addr learned my addr 172.21.15.102:0/3847376734 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.122+0000 7f540fe00640 1 --2- 172.21.15.102:0/3847376734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f65e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f5417400640 1 -- 172.21.15.102:0/3847376734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 msgr2=0x55cc3f65b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f5417400640 1 --2- 172.21.15.102:0/3847376734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f65b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f5417400640 1 -- 172.21.15.102:0/3847376734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 msgr2=0x55cc3f65e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f5417400640 1 --2- 172.21.15.102:0/3847376734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f65e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f5417400640 1 -- 172.21.15.102:0/3847376734 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc3f2cc2d0 con 0x55cc3f6562c0 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f5416a00640 1 --2- 172.21.15.102:0/3847376734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f65b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f540f400640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53fc0a3c40 0x7f53fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f5417400640 1 --2- 172.21.15.102:0/3847376734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6566e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f54100039f0 tx=0x7f5410003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c74fc5bb6f4177a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f540f400640 1 -- 172.21.15.102:0/3800495906 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f54080d0d50 con 0x7f5418104990 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f5416a00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53fc0a3c40 0x7f53fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.123+0000 7f5407e00640 1 -- 172.21.15.102:0/3847376734 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541001a470 con 0x55cc3f6562c0 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f5407e00640 1 -- 172.21.15.102:0/3847376734 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5410015070 con 0x55cc3f6562c0 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f5407e00640 1 -- 172.21.15.102:0/3847376734 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54100107e0 con 0x55cc3f6562c0 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f54205fe740 1 -- 172.21.15.102:0/3847376734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 msgr2=0x55cc3f6566e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f5416a00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53fc0a3c40 0x7f53fc0a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55cc3f657720 tx=0x7f54080d48a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f54205fe740 1 --2- 172.21.15.102:0/3847376734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6566e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f54100039f0 tx=0x7f5410003eb0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f54205fe740 1 -- 172.21.15.102:0/3847376734 shutdown_connections 2024-04-26T20:34:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f54205fe740 1 --2- 172.21.15.102:0/3847376734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f65e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f54205fe740 1 --2- 172.21.15.102:0/3847376734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f65b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f54205fe740 1 --2- 172.21.15.102:0/3847376734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6566e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f54205fe740 1 -- 172.21.15.102:0/3847376734 >> 172.21.15.102:0/3847376734 conn(0x55cc3f5ead10 msgr2=0x55cc3f651d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f54205fe740 1 -- 172.21.15.102:0/3847376734 shutdown_connections 2024-04-26T20:34:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.124+0000 7f54205fe740 1 -- 172.21.15.102:0/3847376734 wait complete. 2024-04-26T20:34:35.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.125+0000 7f54205fe740 1 -- start start 2024-04-26T20:34:35.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.125+0000 7f54205fe740 1 Processor -- start 2024-04-26T20:34:35.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.125+0000 7f54205fe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6606e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.125+0000 7f54205fe740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f660c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f54205fe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f667c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f54205fe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f65edd0 con 0x55cc3f65be20 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f54205fe740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f541f70 con 0x55cc3f656cb0 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f54205fe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f65ef50 con 0x55cc3f6562c0 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f54205fe740 10 client.0 _ll_register_callbacks cb 0x55cc3f5b0480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f5417400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f5417400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6606e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55632/0 (socket says 172.21.15.102:55632) 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f5417400640 1 -- 172.21.15.102:0/2932742870 learned_addr learned my addr 172.21.15.102:0/2932742870 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f5417400640 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f667c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f540fe00640 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f660c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f5417400640 1 -- 172.21.15.102:0/2932742870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 msgr2=0x55cc3f660c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f5417400640 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f660c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f5417400640 1 -- 172.21.15.102:0/2932742870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 msgr2=0x55cc3f667c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f5417400640 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f667c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.126+0000 7f5417400640 1 -- 172.21.15.102:0/2932742870 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc3f3966b0 con 0x55cc3f6562c0 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.127+0000 7f5417400640 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6606e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5410000f80 tx=0x7f5410003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f74cdcc15132a43d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.127+0000 7f5404c00640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.127+0000 7f5404c00640 1 -- 172.21.15.102:0/2932742870 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541002e060 con 0x55cc3f6562c0 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.127+0000 7f54205fe740 1 -- 172.21.15.102:0/2932742870 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5410015070 con 0x55cc3f6562c0 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.127+0000 7f5404c00640 1 -- 172.21.15.102:0/2932742870 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f5410010580 con 0x55cc3f6562c0 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.127+0000 7f5404c00640 1 -- 172.21.15.102:0/2932742870 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5410035050 con 0x55cc3f6562c0 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.127+0000 7f54205fe740 10 client.25180 fetch_fsmap learned FSMap version 27 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.127+0000 7f54205fe740 1 -- 172.21.15.102:0/2932742870 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cc3f66ac90 con 0x55cc3f6562c0 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.127+0000 7f5404c00640 1 -- 172.21.15.102:0/2932742870 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5410015070 con 0x55cc3f6562c0 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.128+0000 7f5404c00640 1 -- 172.21.15.102:0/2932742870 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f541001a6d0 con 0x55cc3f6562c0 2024-04-26T20:34:35.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.128+0000 7f54205fe740 10 client.25180 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.128+0000 7f54205fe740 10 client.25180 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.128+0000 7f54205fe740 20 client.25180 populate_metadata read hostname 'smithi102' 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.128+0000 7f54205fe740 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x55cc3f66cea0 0x55cc3f66e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.128+0000 7f54205fe740 4 client.25180 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.128+0000 7f54205fe740 1 -- 172.21.15.102:0/2932742870 --> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc3f5c8210 con 0x55cc3f66cea0 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.128+0000 7f540fe00640 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x55cc3f66cea0 0x55cc3f66e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.129+0000 7f540fe00640 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x55cc3f66cea0 0x55cc3f66e310 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dbbadac3a1f8fa89 server_cookie=2e48daa7a8f39db1 in_seq=0 out_seq=0 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.129+0000 7f5404c00640 10 client.25180 ms_handle_connect on v2:172.21.15.112:6836/82964069 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f5404c00640 1 -- 172.21.15.102:0/2932742870 <== mds.3 v2:172.21.15.112:6836/82964069 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc3f5c8210 con 0x55cc3f66cea0 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f5404c00640 10 client.25180 handle_command_reply: tid=0 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 client.25180 shutdown 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 -- 172.21.15.102:0/2932742870 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 msgr2=0x55cc3f6606e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6606e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5410000f80 tx=0x7f5410003eb0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 -- 172.21.15.102:0/2932742870 shutdown_connections 2024-04-26T20:34:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x55cc3f66cea0 0x55cc3f66e310 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f667c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f660c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 --2- 172.21.15.102:0/2932742870 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f6606e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f5404c00640 0 client.25180 ms_handle_reset on v2:172.21.15.112:6836/82964069 2024-04-26T20:34:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 -- 172.21.15.102:0/2932742870 >> 172.21.15.102:0/2932742870 conn(0x55cc3f5ead10 msgr2=0x55cc3f650f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 -- 172.21.15.102:0/2932742870 shutdown_connections 2024-04-26T20:34:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 1 -- 172.21.15.102:0/2932742870 wait complete. 2024-04-26T20:34:35.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.130+0000 7f54205fe740 20 client.25180 trim_cache size 0 max 16384 2024-04-26T20:34:35.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.149+0000 7f54205fe740 1 Processor -- start 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.149+0000 7f54205fe740 1 -- start start 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.149+0000 7f54205fe740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f6562c0 0x55cc3f6602f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.149+0000 7f54205fe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f656cb0 0x55cc3f667c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.149+0000 7f54205fe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f65be20 0x55cc3f660830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.149+0000 7f54205fe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f66c650 con 0x55cc3f656cb0 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.149+0000 7f54205fe740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f66cd20 con 0x55cc3f6562c0 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.149+0000 7f54205fe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f66d3f0 con 0x55cc3f65be20 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f5417400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f6562c0 0x55cc3f6602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f5417400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f6562c0 0x55cc3f6602f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:45700/0 (socket says 172.21.15.102:45700) 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f5417400640 1 -- 172.21.15.102:0/1910998245 learned_addr learned my addr 172.21.15.102:0/1910998245 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f5417400640 1 --2- 172.21.15.102:0/1910998245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f65be20 0x55cc3f660830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f540fe00640 1 --2- 172.21.15.102:0/1910998245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f656cb0 0x55cc3f667c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f5417400640 1 -- 172.21.15.102:0/1910998245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f65be20 msgr2=0x55cc3f660830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f5417400640 1 --2- 172.21.15.102:0/1910998245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f65be20 0x55cc3f660830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f5417400640 1 -- 172.21.15.102:0/1910998245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f656cb0 msgr2=0x55cc3f667c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f5417400640 1 --2- 172.21.15.102:0/1910998245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f656cb0 0x55cc3f667c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f5417400640 1 -- 172.21.15.102:0/1910998245 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc3f65b360 con 0x55cc3f6562c0 2024-04-26T20:34:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f540fe00640 1 --2- 172.21.15.102:0/1910998245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f656cb0 0x55cc3f667c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T20:34:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.150+0000 7f5417400640 1 --2- 172.21.15.102:0/1910998245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f6562c0 0x55cc3f6602f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f54100047b0 tx=0x7f5410004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6888cca3b844853b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f5407400640 1 -- 172.21.15.102:0/1910998245 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541002e040 con 0x55cc3f6562c0 2024-04-26T20:34:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f5407400640 1 -- 172.21.15.102:0/1910998245 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5410015070 con 0x55cc3f6562c0 2024-04-26T20:34:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f5407400640 1 -- 172.21.15.102:0/1910998245 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410004be0 con 0x55cc3f6562c0 2024-04-26T20:34:35.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f54205fe740 1 -- 172.21.15.102:0/1910998245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f6562c0 msgr2=0x55cc3f6602f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f54205fe740 1 --2- 172.21.15.102:0/1910998245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f6562c0 0x55cc3f6602f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f54100047b0 tx=0x7f5410004740 comp rx=0 tx=0).stop 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f54205fe740 1 -- 172.21.15.102:0/1910998245 shutdown_connections 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f54205fe740 1 --2- 172.21.15.102:0/1910998245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f656cb0 0x55cc3f667c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f54205fe740 1 --2- 172.21.15.102:0/1910998245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f65be20 0x55cc3f660830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f54205fe740 1 --2- 172.21.15.102:0/1910998245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f6562c0 0x55cc3f6602f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f54205fe740 1 -- 172.21.15.102:0/1910998245 >> 172.21.15.102:0/1910998245 conn(0x55cc3f5ead10 msgr2=0x55cc3f6580b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f54205fe740 1 -- 172.21.15.102:0/1910998245 shutdown_connections 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.151+0000 7f54205fe740 1 -- 172.21.15.102:0/1910998245 wait complete. 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.152+0000 7f54205fe740 1 -- start start 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.152+0000 7f54205fe740 1 Processor -- start 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.152+0000 7f54205fe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f663e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.152+0000 7f54205fe740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f664390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.152+0000 7f54205fe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f667050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.152+0000 7f54205fe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f66e190 con 0x55cc3f65be20 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.152+0000 7f54205fe740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f66dac0 con 0x55cc3f656cb0 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.152+0000 7f54205fe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cc3f66e860 con 0x55cc3f6562c0 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.152+0000 7f54205fe740 10 client.0 _ll_register_callbacks cb 0x7f5408005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.152+0000 7f5417400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f663e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f5417400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f663e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55644/0 (socket says 172.21.15.102:55644) 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f5417400640 1 -- 172.21.15.102:0/2396929776 learned_addr learned my addr 172.21.15.102:0/2396929776 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f540fe00640 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f664390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f5417400640 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f667050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f540fe00640 1 -- 172.21.15.102:0/2396929776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 msgr2=0x55cc3f663e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f540fe00640 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f663e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f540fe00640 1 -- 172.21.15.102:0/2396929776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 msgr2=0x55cc3f667050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f540fe00640 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f667050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f540fe00640 1 -- 172.21.15.102:0/2396929776 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc3f667640 con 0x55cc3f656cb0 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f540fe00640 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f664390 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f540002a9e0 tx=0x7f540002aaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=951cbb334e1bdbe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f5404c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f5404c00640 1 -- 172.21.15.102:0/2396929776 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400049020 con 0x55cc3f656cb0 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f5404c00640 1 -- 172.21.15.102:0/2396929776 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f5400005240 con 0x55cc3f656cb0 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.153+0000 7f54205fe740 1 -- 172.21.15.102:0/2396929776 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5410015070 con 0x55cc3f656cb0 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.154+0000 7f5404c00640 1 -- 172.21.15.102:0/2396929776 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5400054050 con 0x55cc3f656cb0 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.154+0000 7f54205fe740 10 client.25187 fetch_fsmap learned FSMap version 27 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.154+0000 7f54205fe740 1 -- 172.21.15.102:0/2396929776 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cc3f6dbde0 con 0x55cc3f656cb0 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.154+0000 7f5404c00640 1 -- 172.21.15.102:0/2396929776 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5410015070 con 0x55cc3f656cb0 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.154+0000 7f5404c00640 1 -- 172.21.15.102:0/2396929776 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f54000047e0 con 0x55cc3f656cb0 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.154+0000 7f54205fe740 10 client.25187 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.154+0000 7f54205fe740 10 client.25187 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T20:34:35.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.154+0000 7f54205fe740 20 client.25187 populate_metadata read hostname 'smithi102' 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.154+0000 7f54205fe740 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x55cc3f6e09d0 0x55cc3f6df6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.155+0000 7f54205fe740 4 client.25187 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.155+0000 7f54205fe740 1 -- 172.21.15.102:0/2396929776 --> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] -- command(tid 0: {"prefix": "status"}) -- 0x55cc3f5c2080 con 0x55cc3f6e09d0 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.155+0000 7f5417400640 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x55cc3f6e09d0 0x55cc3f6df6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.155+0000 7f5417400640 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x55cc3f6e09d0 0x55cc3f6df6e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=85f48d3b5cf85900 server_cookie=d0171d757ac50a67 in_seq=0 out_seq=0 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.155+0000 7f5404c00640 10 client.25187 ms_handle_connect on v2:172.21.15.112:6836/82964069 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.155+0000 7f5404c00640 1 -- 172.21.15.102:0/2396929776 <== mds.3 v2:172.21.15.112:6836/82964069 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55cc3f5c2080 con 0x55cc3f6e09d0 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.155+0000 7f5404c00640 10 client.25187 handle_command_reply: tid=0 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.155+0000 7f54205fe740 1 client.25187 shutdown 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 1 -- 172.21.15.102:0/2396929776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 msgr2=0x55cc3f664390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f664390 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f540002a9e0 tx=0x7f540002aaf0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 1 -- 172.21.15.102:0/2396929776 shutdown_connections 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x55cc3f6e09d0 0x55cc3f6df6e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cc3f65be20 0x55cc3f667050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cc3f656cb0 0x55cc3f664390 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 1 --2- 172.21.15.102:0/2396929776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc3f6562c0 0x55cc3f663e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f5404c00640 0 client.25187 ms_handle_reset on v2:172.21.15.112:6836/82964069 2024-04-26T20:34:35.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 1 -- 172.21.15.102:0/2396929776 >> 172.21.15.102:0/2396929776 conn(0x55cc3f5ead10 msgr2=0x55cc3f6602f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 1 -- 172.21.15.102:0/2396929776 shutdown_connections 2024-04-26T20:34:35.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 1 -- 172.21.15.102:0/2396929776 wait complete. 2024-04-26T20:34:35.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f54205fe740 20 client.25187 trim_cache size 0 max 16384 2024-04-26T20:34:35.162 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:34:35.162 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:34:35.162 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 3, 2024-04-26T20:34:35.163 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24533, 2024-04-26T20:34:35.163 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:34:35.163 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:34:35.163 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:34:35.163 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 101.489492341, 2024-04-26T20:34:35.163 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 27, 2024-04-26T20:34:35.163 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-04-26T20:34:35.163 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:34:35.163 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 149.304877815 2024-04-26T20:34:35.163 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:34:35.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f5417e00640 1 -- 172.21.15.102:0/3800495906 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53fc0a3c40 msgr2=0x7f53fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f5417e00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53fc0a3c40 0x7f53fc0a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55cc3f657720 tx=0x7f54080d48a0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f5417e00640 1 -- 172.21.15.102:0/3800495906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418104990 msgr2=0x7f541819aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.156+0000 7f5417e00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418104990 0x7f541819aba0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f540800e9c0 tx=0x7f540800ee80 comp rx=0 tx=0).stop 2024-04-26T20:34:35.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.157+0000 7f5417e00640 1 -- 172.21.15.102:0/3800495906 shutdown_connections 2024-04-26T20:34:35.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.157+0000 7f5417e00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f53fc0a3c40 0x7f53fc0a6100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.157+0000 7f5417e00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5418106540 0x7f54181a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.157+0000 7f5417e00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5418105b80 0x7f541819b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.157+0000 7f5417e00640 1 --2- 172.21.15.102:0/3800495906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418104990 0x7f541819aba0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.157+0000 7f5417e00640 1 -- 172.21.15.102:0/3800495906 >> 172.21.15.102:0/3800495906 conn(0x7f54180fff00 msgr2=0x7f5418108fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.157+0000 7f5417e00640 1 -- 172.21.15.102:0/3800495906 shutdown_connections 2024-04-26T20:34:35.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.157+0000 7f5417e00640 1 -- 172.21.15.102:0/3800495906 wait complete. 2024-04-26T20:34:35.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-26T20:34:35.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.355+0000 7f604d000640 1 -- 172.21.15.102:0/4094422734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6048104990 msgr2=0x7f6048104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.355+0000 7f604d000640 1 --2- 172.21.15.102:0/4094422734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6048104990 0x7f6048104db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6038009900 tx=0x7f603802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.355+0000 7f604d000640 1 -- 172.21.15.102:0/4094422734 shutdown_connections 2024-04-26T20:34:35.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.355+0000 7f604d000640 1 --2- 172.21.15.102:0/4094422734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048105b80 0x7f6048106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.355+0000 7f604d000640 1 --2- 172.21.15.102:0/4094422734 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6048106540 0x7f604810cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.355+0000 7f604d000640 1 --2- 172.21.15.102:0/4094422734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6048104990 0x7f6048104db0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.355+0000 7f604d000640 1 -- 172.21.15.102:0/4094422734 >> 172.21.15.102:0/4094422734 conn(0x7f60480fff00 msgr2=0x7f6048102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.355+0000 7f604d000640 1 -- 172.21.15.102:0/4094422734 shutdown_connections 2024-04-26T20:34:35.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.355+0000 7f604d000640 1 -- 172.21.15.102:0/4094422734 wait complete. 2024-04-26T20:34:35.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.356+0000 7f604d000640 1 Processor -- start 2024-04-26T20:34:35.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.356+0000 7f604d000640 1 -- start start 2024-04-26T20:34:35.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f604d000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6048104990 0x7f604819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f604d000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6048105b80 0x7f604819b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f604d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048106540 0x7f60481a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f604d000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6048110510 con 0x7f6048106540 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f604d000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6048110390 con 0x7f6048104990 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f604d000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6048110690 con 0x7f6048105b80 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6046a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6048105b80 0x7f604819b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6047e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048106540 0x7f60481a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6046a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6048105b80 0x7f604819b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55660/0 (socket says 172.21.15.102:55660) 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6046a00640 1 -- 172.21.15.102:0/2227055804 learned_addr learned my addr 172.21.15.102:0/2227055804 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6047400640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6048104990 0x7f604819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6046a00640 1 -- 172.21.15.102:0/2227055804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6048104990 msgr2=0x7f604819ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6046a00640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6048104990 0x7f604819ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6046a00640 1 -- 172.21.15.102:0/2227055804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048106540 msgr2=0x7f60481a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6046a00640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048106540 0x7f60481a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6046a00640 1 -- 172.21.15.102:0/2227055804 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60481a27b0 con 0x7f6048105b80 2024-04-26T20:34:35.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.357+0000 7f6047e00640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048106540 0x7f60481a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:35.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.358+0000 7f6046a00640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6048105b80 0x7f604819b0d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f603400e9f0 tx=0x7f603400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.358+0000 7f603fe00640 1 -- 172.21.15.102:0/2227055804 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603400cd80 con 0x7f6048105b80 2024-04-26T20:34:35.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.358+0000 7f603fe00640 1 -- 172.21.15.102:0/2227055804 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60340108e0 con 0x7f6048105b80 2024-04-26T20:34:35.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.358+0000 7f603fe00640 1 -- 172.21.15.102:0/2227055804 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6034018980 con 0x7f6048105b80 2024-04-26T20:34:35.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.358+0000 7f604d000640 1 -- 172.21.15.102:0/2227055804 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60481a2aa0 con 0x7f6048105b80 2024-04-26T20:34:35.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.358+0000 7f604d000640 1 -- 172.21.15.102:0/2227055804 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60481a3040 con 0x7f6048105b80 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.361+0000 7f60513c5740 1 Processor -- start 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.361+0000 7f60513c5740 1 -- start start 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.361+0000 7f60513c5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc934602c0 0x55fc934606e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.361+0000 7f60513c5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93460cb0 0x55fc934657b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.361+0000 7f60513c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc93465e20 0x55fc934682b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.361+0000 7f60513c5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55fc933c4600 con 0x55fc93460cb0 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.361+0000 7f60513c5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fc93361100 con 0x55fc934602c0 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.361+0000 7f60513c5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55fc933c05c0 con 0x55fc93465e20 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.361+0000 7f6047400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93460cb0 0x55fc934657b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f6047e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc934602c0 0x55fc934606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f6047400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93460cb0 0x55fc934657b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37962/0 (socket says 172.21.15.102:37962) 2024-04-26T20:34:35.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f6047400640 1 -- 172.21.15.102:0/1558665838 learned_addr learned my addr 172.21.15.102:0/1558665838 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f603fe00640 1 -- 172.21.15.102:0/2227055804 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f6034010430 con 0x7f6048105b80 2024-04-26T20:34:35.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f6047e00640 1 -- 172.21.15.102:0/1558665838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc93465e20 msgr2=0x55fc934682b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T20:34:35.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f6047e00640 1 --2- 172.21.15.102:0/1558665838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc93465e20 0x55fc934682b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f6047e00640 1 -- 172.21.15.102:0/1558665838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93460cb0 msgr2=0x55fc934657b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f6047e00640 1 --2- 172.21.15.102:0/1558665838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93460cb0 0x55fc934657b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f6047e00640 1 -- 172.21.15.102:0/1558665838 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc930d62d0 con 0x55fc934602c0 2024-04-26T20:34:35.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f6047400640 1 --2- 172.21.15.102:0/1558665838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93460cb0 0x55fc934657b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:35.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.362+0000 7f6047e00640 1 --2- 172.21.15.102:0/1558665838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc934602c0 0x55fc934606e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6040005fd0 tx=0x7f6040011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abee6621312f2616 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f603cc00640 1 -- 172.21.15.102:0/1558665838 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040004d20 con 0x55fc934602c0 2024-04-26T20:34:35.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f603cc00640 1 -- 172.21.15.102:0/1558665838 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f604000f9a0 con 0x55fc934602c0 2024-04-26T20:34:35.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f603cc00640 1 -- 172.21.15.102:0/1558665838 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040004430 con 0x55fc934602c0 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f60513c5740 1 -- 172.21.15.102:0/1558665838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc934602c0 msgr2=0x55fc934606e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f60513c5740 1 --2- 172.21.15.102:0/1558665838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc934602c0 0x55fc934606e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6040005fd0 tx=0x7f6040011bf0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f60513c5740 1 -- 172.21.15.102:0/1558665838 shutdown_connections 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f60513c5740 1 --2- 172.21.15.102:0/1558665838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93460cb0 0x55fc934657b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f60513c5740 1 --2- 172.21.15.102:0/1558665838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc93465e20 0x55fc934682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f60513c5740 1 --2- 172.21.15.102:0/1558665838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc934602c0 0x55fc934606e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f60513c5740 1 -- 172.21.15.102:0/1558665838 >> 172.21.15.102:0/1558665838 conn(0x55fc933f4d10 msgr2=0x55fc9345bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f60513c5740 1 -- 172.21.15.102:0/1558665838 shutdown_connections 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.363+0000 7f60513c5740 1 -- 172.21.15.102:0/1558665838 wait complete. 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.364+0000 7f603fe00640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bd0 0x7f602c0a6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.364+0000 7f603fe00640 1 -- 172.21.15.102:0/2227055804 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f60340d0ba0 con 0x7f6048105b80 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.364+0000 7f6047400640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bd0 0x7f602c0a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.364+0000 7f60513c5740 1 -- start start 2024-04-26T20:34:35.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.364+0000 7f60513c5740 1 Processor -- start 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f6047400640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bd0 0x7f602c0a6090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55fc93461720 tx=0x7f6038005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f60513c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc934602c0 0x55fc9346a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f60513c5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 0x55fc9346ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f60513c5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc93471c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f60513c5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55fc93468dd0 con 0x55fc93465e20 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f60513c5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fc9334bf70 con 0x55fc93460cb0 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f60513c5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55fc93468f50 con 0x55fc934602c0 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f60513c5740 10 client.0 _ll_register_callbacks cb 0x55fc933ba480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f6047e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc93471c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f6047e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc93471c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37964/0 (socket says 172.21.15.102:37964) 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f6047e00640 1 -- 172.21.15.102:0/1061821613 learned_addr learned my addr 172.21.15.102:0/1061821613 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f6047400640 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc934602c0 0x55fc9346a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f6046a00640 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 0x55fc9346ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f6047e00640 1 -- 172.21.15.102:0/1061821613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc934602c0 msgr2=0x55fc9346a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f6047e00640 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc934602c0 0x55fc9346a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f6047e00640 1 -- 172.21.15.102:0/1061821613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 msgr2=0x55fc9346ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.365+0000 7f6047e00640 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 0x55fc9346ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.366+0000 7f6047e00640 1 -- 172.21.15.102:0/1061821613 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc931a06b0 con 0x55fc93465e20 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.366+0000 7f6047e00640 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc93471c60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f604000a010 tx=0x7f604000fed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f1783ed058d1ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.366+0000 7f6031600640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.366+0000 7f6031600640 1 -- 172.21.15.102:0/1061821613 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040028020 con 0x55fc93465e20 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.366+0000 7f6031600640 1 -- 172.21.15.102:0/1061821613 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f60400041d0 con 0x55fc93465e20 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.366+0000 7f60513c5740 1 -- 172.21.15.102:0/1061821613 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f604000f9a0 con 0x55fc93465e20 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.366+0000 7f6031600640 1 -- 172.21.15.102:0/1061821613 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6040033050 con 0x55fc93465e20 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.366+0000 7f60513c5740 10 client.15075 fetch_fsmap learned FSMap version 27 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.366+0000 7f60513c5740 1 -- 172.21.15.102:0/1061821613 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fc93474c90 con 0x55fc93465e20 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.367+0000 7f6031600640 1 -- 172.21.15.102:0/1061821613 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f604000f9a0 con 0x55fc93465e20 2024-04-26T20:34:35.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.367+0000 7f6031600640 1 -- 172.21.15.102:0/1061821613 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6040018070 con 0x55fc93465e20 2024-04-26T20:34:35.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.367+0000 7f60513c5740 10 client.15075 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:34:35.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.367+0000 7f60513c5740 10 client.15075 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T20:34:35.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.367+0000 7f60513c5740 20 client.15075 populate_metadata read hostname 'smithi102' 2024-04-26T20:34:35.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.367+0000 7f60513c5740 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x55fc934770b0 0x55fc93478520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.367+0000 7f60513c5740 4 client.15075 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:34:35.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.367+0000 7f60513c5740 1 -- 172.21.15.102:0/1061821613 --> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc933d2210 con 0x55fc934770b0 2024-04-26T20:34:35.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.367+0000 7f6047400640 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x55fc934770b0 0x55fc93478520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.368+0000 7f6047400640 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x55fc934770b0 0x55fc93478520 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7b8ba4382b6828ce server_cookie=9f47ac1037c69d32 in_seq=0 out_seq=0 2024-04-26T20:34:35.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.368+0000 7f6031600640 10 client.15075 ms_handle_connect on v2:172.21.15.130:6838/1700008514 2024-04-26T20:34:35.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f6031600640 1 -- 172.21.15.102:0/1061821613 <== mds.2 v2:172.21.15.130:6838/1700008514 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc933d2210 con 0x55fc934770b0 2024-04-26T20:34:35.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f6031600640 10 client.15075 handle_command_reply: tid=0 2024-04-26T20:34:35.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 client.15075 shutdown 2024-04-26T20:34:35.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 -- 172.21.15.102:0/1061821613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 msgr2=0x55fc93471c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc93471c60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f604000a010 tx=0x7f604000fed0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 -- 172.21.15.102:0/1061821613 shutdown_connections 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x55fc934770b0 0x55fc93478520 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc93471c60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 0x55fc9346ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f6031600640 0 client.15075 ms_handle_reset on v2:172.21.15.130:6838/1700008514 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 --2- 172.21.15.102:0/1061821613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc934602c0 0x55fc9346a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 -- 172.21.15.102:0/1061821613 >> 172.21.15.102:0/1061821613 conn(0x55fc933f4d10 msgr2=0x55fc9345af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 -- 172.21.15.102:0/1061821613 shutdown_connections 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.369+0000 7f60513c5740 1 -- 172.21.15.102:0/1061821613 wait complete. 2024-04-26T20:34:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.370+0000 7f60513c5740 20 client.15075 trim_cache size 0 max 16384 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f60513c5740 1 Processor -- start 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f60513c5740 1 -- start start 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f60513c5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc934602c0 0x55fc934620b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f60513c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc93460cb0 0x55fc934625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f60513c5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93465e20 0x55fc93471b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f60513c5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55fc93476860 con 0x55fc934602c0 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f60513c5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fc93476f30 con 0x55fc93465e20 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f60513c5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55fc93477600 con 0x55fc93460cb0 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f6047400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc934602c0 0x55fc934620b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f6047400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc934602c0 0x55fc934620b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37970/0 (socket says 172.21.15.102:37970) 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.389+0000 7f6047400640 1 -- 172.21.15.102:0/117834785 learned_addr learned my addr 172.21.15.102:0/117834785 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6047e00640 1 --2- 172.21.15.102:0/117834785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93465e20 0x55fc93471b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6046a00640 1 --2- 172.21.15.102:0/117834785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc93460cb0 0x55fc934625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6047400640 1 -- 172.21.15.102:0/117834785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc93460cb0 msgr2=0x55fc934625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6047400640 1 --2- 172.21.15.102:0/117834785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc93460cb0 0x55fc934625f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6047400640 1 -- 172.21.15.102:0/117834785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93465e20 msgr2=0x55fc93471b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6047400640 1 --2- 172.21.15.102:0/117834785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93465e20 0x55fc93471b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6047400640 1 -- 172.21.15.102:0/117834785 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc93465380 con 0x55fc934602c0 2024-04-26T20:34:35.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6047400640 1 --2- 172.21.15.102:0/117834785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc934602c0 0x55fc934620b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6038004580 tx=0x7f6038040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf8e958555d119c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6033e00640 1 -- 172.21.15.102:0/117834785 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6038035040 con 0x55fc934602c0 2024-04-26T20:34:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6033e00640 1 -- 172.21.15.102:0/117834785 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60380548d0 con 0x55fc934602c0 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.390+0000 7f6033e00640 1 -- 172.21.15.102:0/117834785 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603803d450 con 0x55fc934602c0 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 -- 172.21.15.102:0/117834785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc934602c0 msgr2=0x55fc934620b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 --2- 172.21.15.102:0/117834785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc934602c0 0x55fc934620b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6038004580 tx=0x7f6038040000 comp rx=0 tx=0).stop 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 -- 172.21.15.102:0/117834785 shutdown_connections 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 --2- 172.21.15.102:0/117834785 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93465e20 0x55fc93471b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 --2- 172.21.15.102:0/117834785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc93460cb0 0x55fc934625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 --2- 172.21.15.102:0/117834785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc934602c0 0x55fc934620b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 -- 172.21.15.102:0/117834785 >> 172.21.15.102:0/117834785 conn(0x55fc933f4d10 msgr2=0x55fc93476090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 -- 172.21.15.102:0/117834785 shutdown_connections 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 -- 172.21.15.102:0/117834785 wait complete. 2024-04-26T20:34:35.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 -- start start 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.391+0000 7f60513c5740 1 Processor -- start 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f60513c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc934602c0 0x55fc9346fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f60513c5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 0x55fc934e1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f60513c5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc934e3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f60513c5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55fc934783a0 con 0x55fc93465e20 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f60513c5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fc93477cd0 con 0x55fc93460cb0 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f60513c5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55fc93478a70 con 0x55fc934602c0 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f60513c5740 10 client.0 _ll_register_callbacks cb 0x7f60400093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f6047e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc934e3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f6046a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 0x55fc934e1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f6047e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc934e3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37982/0 (socket says 172.21.15.102:37982) 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f6047e00640 1 -- 172.21.15.102:0/2320574654 learned_addr learned my addr 172.21.15.102:0/2320574654 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.392+0000 7f6047400640 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc934602c0 0x55fc9346fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6046a00640 1 -- 172.21.15.102:0/2320574654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc934602c0 msgr2=0x55fc9346fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6046a00640 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc934602c0 0x55fc9346fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6046a00640 1 -- 172.21.15.102:0/2320574654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 msgr2=0x55fc934e3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6046a00640 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc934e3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6046a00640 1 -- 172.21.15.102:0/2320574654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc934e44b0 con 0x55fc93460cb0 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6047e00640 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc934e3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6046a00640 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 0x55fc934e1850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f60340ce1a0 tx=0x7f603400cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99b2ece27cf2ca2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6031600640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6031600640 1 -- 172.21.15.102:0/2320574654 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603409ca20 con 0x55fc93460cb0 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6031600640 1 -- 172.21.15.102:0/2320574654 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f60340bbb30 con 0x55fc93460cb0 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f60513c5740 1 -- 172.21.15.102:0/2320574654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60380548d0 con 0x55fc93460cb0 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f6031600640 1 -- 172.21.15.102:0/2320574654 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f60340a0d20 con 0x55fc93460cb0 2024-04-26T20:34:35.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f60513c5740 10 client.25205 fetch_fsmap learned FSMap version 27 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.393+0000 7f60513c5740 1 -- 172.21.15.102:0/2320574654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fc934e5820 con 0x55fc93460cb0 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.394+0000 7f6031600640 1 -- 172.21.15.102:0/2320574654 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60380548d0 con 0x55fc93460cb0 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.394+0000 7f6031600640 1 -- 172.21.15.102:0/2320574654 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6034098770 con 0x55fc93460cb0 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.394+0000 7f60513c5740 10 client.25205 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.394+0000 7f60513c5740 10 client.25205 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.394+0000 7f60513c5740 20 client.25205 populate_metadata read hostname 'smithi102' 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.394+0000 7f60513c5740 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x55fc934ea3d0 0x55fc934e90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.394+0000 7f60513c5740 4 client.25205 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.394+0000 7f60513c5740 1 -- 172.21.15.102:0/2320574654 --> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] -- command(tid 0: {"prefix": "status"}) -- 0x55fc933cc080 con 0x55fc934ea3d0 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.394+0000 7f6047400640 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x55fc934ea3d0 0x55fc934e90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f6047400640 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x55fc934ea3d0 0x55fc934e90e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bfeee43fb9dd8c91 server_cookie=a6049d8d59f7693d in_seq=0 out_seq=0 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f6031600640 10 client.25205 ms_handle_connect on v2:172.21.15.130:6838/1700008514 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f6031600640 1 -- 172.21.15.102:0/2320574654 <== mds.2 v2:172.21.15.130:6838/1700008514 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55fc933cc080 con 0x55fc934ea3d0 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f6031600640 10 client.25205 handle_command_reply: tid=0 2024-04-26T20:34:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 client.25205 shutdown 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 -- 172.21.15.102:0/2320574654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 msgr2=0x55fc934e1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 0x55fc934e1850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f60340ce1a0 tx=0x7f603400cf20 comp rx=0 tx=0).stop 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 -- 172.21.15.102:0/2320574654 shutdown_connections 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x55fc934ea3d0 0x55fc934e90e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fc93465e20 0x55fc934e3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc93460cb0 0x55fc934e1850 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f6031600640 0 client.25205 ms_handle_reset on v2:172.21.15.130:6838/1700008514 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 --2- 172.21.15.102:0/2320574654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fc934602c0 0x55fc9346fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 -- 172.21.15.102:0/2320574654 >> 172.21.15.102:0/2320574654 conn(0x55fc933f4d10 msgr2=0x55fc934620b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 -- 172.21.15.102:0/2320574654 shutdown_connections 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 1 -- 172.21.15.102:0/2320574654 wait complete. 2024-04-26T20:34:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.395+0000 7f60513c5740 20 client.25205 trim_cache size 0 max 16384 2024-04-26T20:34:35.406 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:34:35.406 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:34:35.406 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 2, 2024-04-26T20:34:35.406 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24547, 2024-04-26T20:34:35.406 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:34:35.406 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:34:35.406 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:34:35.406 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 102.740909277, 2024-04-26T20:34:35.407 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 27, 2024-04-26T20:34:35.407 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-04-26T20:34:35.407 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:34:35.407 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 146.68679202600001 2024-04-26T20:34:35.407 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 -- 172.21.15.102:0/2227055804 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bd0 msgr2=0x7f602c0a6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bd0 0x7f602c0a6090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55fc93461720 tx=0x7f6038005a20 comp rx=0 tx=0).stop 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 -- 172.21.15.102:0/2227055804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6048105b80 msgr2=0x7f604819b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6048105b80 0x7f604819b0d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f603400e9f0 tx=0x7f603400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 -- 172.21.15.102:0/2227055804 shutdown_connections 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f602c0a3bd0 0x7f602c0a6090 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6048106540 0x7f60481a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6048105b80 0x7f604819b0d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 --2- 172.21.15.102:0/2227055804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6048104990 0x7f604819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 -- 172.21.15.102:0/2227055804 >> 172.21.15.102:0/2227055804 conn(0x7f60480fff00 msgr2=0x7f6048108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 -- 172.21.15.102:0/2227055804 shutdown_connections 2024-04-26T20:34:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.396+0000 7f604d000640 1 -- 172.21.15.102:0/2227055804 wait complete. 2024-04-26T20:34:35.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-26T20:34:35.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.612+0000 7fc1f3000640 1 -- 172.21.15.102:0/2398913025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ec101660 msgr2=0x7fc1ec101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.612+0000 7fc1f3000640 1 --2- 172.21.15.102:0/2398913025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ec101660 0x7fc1ec101a80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009900 tx=0x7fc1e002a410 comp rx=0 tx=0).stop 2024-04-26T20:34:35.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.613+0000 7fc1f3000640 1 -- 172.21.15.102:0/2398913025 shutdown_connections 2024-04-26T20:34:35.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.613+0000 7fc1f3000640 1 --2- 172.21.15.102:0/2398913025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1ec103210 0x7fc1ec109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.613+0000 7fc1f3000640 1 --2- 172.21.15.102:0/2398913025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec102850 0x7fc1ec102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.613+0000 7fc1f3000640 1 --2- 172.21.15.102:0/2398913025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ec101660 0x7fc1ec101a80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.613+0000 7fc1f3000640 1 -- 172.21.15.102:0/2398913025 >> 172.21.15.102:0/2398913025 conn(0x7fc1ec0fcbb0 msgr2=0x7fc1ec0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.613+0000 7fc1f3000640 1 -- 172.21.15.102:0/2398913025 shutdown_connections 2024-04-26T20:34:35.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.613+0000 7fc1f3000640 1 -- 172.21.15.102:0/2398913025 wait complete. 2024-04-26T20:34:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.614+0000 7fc1f3000640 1 Processor -- start 2024-04-26T20:34:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.614+0000 7fc1f3000640 1 -- start start 2024-04-26T20:34:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.614+0000 7fc1f3000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ec101660 0x7fc1ec19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.614+0000 7fc1f3000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1ec102850 0x7fc1ec19b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.614+0000 7fc1f3000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec103210 0x7fc1ec1a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.614+0000 7fc1f3000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec10d1e0 con 0x7fc1ec101660 2024-04-26T20:34:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.614+0000 7fc1f3000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec10d060 con 0x7fc1ec103210 2024-04-26T20:34:35.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.614+0000 7fc1f3000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec10d360 con 0x7fc1ec102850 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.614+0000 7fc1f1c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ec101660 0x7fc1ec19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.614+0000 7fc1f1c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ec101660 0x7fc1ec19abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38002/0 (socket says 172.21.15.102:38002) 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.615+0000 7fc1f1c00640 1 -- 172.21.15.102:0/3568637132 learned_addr learned my addr 172.21.15.102:0/3568637132 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.615+0000 7fc1f2600640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec103210 0x7fc1ec1a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.615+0000 7fc1f1200640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1ec102850 0x7fc1ec19b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.615+0000 7fc1f1c00640 1 -- 172.21.15.102:0/3568637132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1ec102850 msgr2=0x7fc1ec19b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.615+0000 7fc1f1c00640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1ec102850 0x7fc1ec19b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.615+0000 7fc1f1c00640 1 -- 172.21.15.102:0/3568637132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec103210 msgr2=0x7fc1ec1a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.615+0000 7fc1f1c00640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec103210 0x7fc1ec1a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.615+0000 7fc1f1c00640 1 -- 172.21.15.102:0/3568637132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1ec1a27c0 con 0x7fc1ec101660 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.615+0000 7fc1f1c00640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ec101660 0x7fc1ec19abf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0031500 tx=0x7fc1e0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.616+0000 7fc1ea000640 1 -- 172.21.15.102:0/3568637132 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e003f070 con 0x7fc1ec101660 2024-04-26T20:34:35.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.616+0000 7fc1ea000640 1 -- 172.21.15.102:0/3568637132 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1e0031c50 con 0x7fc1ec101660 2024-04-26T20:34:35.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.616+0000 7fc1f3000640 1 -- 172.21.15.102:0/3568637132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1ec1a2a50 con 0x7fc1ec101660 2024-04-26T20:34:35.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.616+0000 7fc1ea000640 1 -- 172.21.15.102:0/3568637132 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0035030 con 0x7fc1ec101660 2024-04-26T20:34:35.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.616+0000 7fc1f3000640 1 -- 172.21.15.102:0/3568637132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1ec1a2f90 con 0x7fc1ec101660 2024-04-26T20:34:35.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.619+0000 7fc1ea000640 1 -- 172.21.15.102:0/3568637132 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc1e0042070 con 0x7fc1ec101660 2024-04-26T20:34:35.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.619+0000 7fc1f6ffd740 1 Processor -- start 2024-04-26T20:34:35.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.619+0000 7fc1f6ffd740 1 -- start start 2024-04-26T20:34:35.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.619+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d8704296e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.619+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d87042e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.619+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d8704312b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.619+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d87038d600 con 0x55d87042ee20 2024-04-26T20:34:35.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.619+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d87032a100 con 0x55d870429cb0 2024-04-26T20:34:35.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.619+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d8703895c0 con 0x55d8704292c0 2024-04-26T20:34:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.620+0000 7fc1f2600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d8704296e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.620+0000 7fc1f1200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d8704312b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.620+0000 7fc1f1c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d87042e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.620+0000 7fc1f2600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d8704296e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55710/0 (socket says 172.21.15.102:55710) 2024-04-26T20:34:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.620+0000 7fc1f2600640 1 -- 172.21.15.102:0/3389180534 learned_addr learned my addr 172.21.15.102:0/3389180534 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.620+0000 7fc1f2600640 1 -- 172.21.15.102:0/3389180534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 msgr2=0x55d87042e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.620+0000 7fc1f2600640 1 --2- 172.21.15.102:0/3389180534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d87042e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.620+0000 7fc1f2600640 1 -- 172.21.15.102:0/3389180534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 msgr2=0x55d8704312b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.620+0000 7fc1f2600640 1 --2- 172.21.15.102:0/3389180534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d8704312b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.620+0000 7fc1f2600640 1 -- 172.21.15.102:0/3389180534 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d87009f2d0 con 0x55d8704292c0 2024-04-26T20:34:35.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.621+0000 7fc1f2600640 1 --2- 172.21.15.102:0/3389180534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d8704296e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009fd0 tx=0x7fc1e40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df760ce280bb16ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.621+0000 7fc1ea000640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc1d40a3cc0 0x7fc1d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.621+0000 7fc1dea00640 1 -- 172.21.15.102:0/3389180534 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4004d20 con 0x55d8704292c0 2024-04-26T20:34:35.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.621+0000 7fc1dea00640 1 -- 172.21.15.102:0/3389180534 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1e4004050 con 0x55d8704292c0 2024-04-26T20:34:35.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.621+0000 7fc1dea00640 1 -- 172.21.15.102:0/3389180534 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4005280 con 0x55d8704292c0 2024-04-26T20:34:35.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.621+0000 7fc1ea000640 1 -- 172.21.15.102:0/3568637132 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc1e00f0fe0 con 0x7fc1ec101660 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.621+0000 7fc1f1c00640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc1d40a3cc0 0x7fc1d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/3389180534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 msgr2=0x55d8704296e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f1c00640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc1d40a3cc0 0x7fc1d40a6180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55d87042a720 tx=0x7fc1e00f4af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/3389180534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d8704296e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009fd0 tx=0x7fc1e40118d0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/3389180534 shutdown_connections 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/3389180534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d8704312b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/3389180534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d87042e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/3389180534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d8704296e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/3389180534 >> 172.21.15.102:0/3389180534 conn(0x55d8703bdd10 msgr2=0x55d870424d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/3389180534 shutdown_connections 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/3389180534 wait complete. 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.622+0000 7fc1f6ffd740 1 -- start start 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f6ffd740 1 Processor -- start 2024-04-26T20:34:35.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d870433720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d870433c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d87043aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d870431dd0 con 0x55d87042ee20 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d870314f70 con 0x55d870429cb0 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d870431f50 con 0x55d8704292c0 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f6ffd740 10 client.0 _ll_register_callbacks cb 0x55d870383480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f2600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d870433720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f1200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d870433c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f2600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d870433720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55714/0 (socket says 172.21.15.102:55714) 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f2600640 1 -- 172.21.15.102:0/746859333 learned_addr learned my addr 172.21.15.102:0/746859333 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.623+0000 7fc1f2600640 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d87043aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1f1200640 1 -- 172.21.15.102:0/746859333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 msgr2=0x55d870433720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1f1200640 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d870433720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1f1200640 1 -- 172.21.15.102:0/746859333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 msgr2=0x55d87043aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1f1200640 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d87043aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1f1200640 1 -- 172.21.15.102:0/746859333 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8701696b0 con 0x55d870429cb0 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1f1200640 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d870433c60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec19c0b0 tx=0x7fc1d8012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9ac28fa44bed38e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1d3400640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1d3400640 1 -- 172.21.15.102:0/746859333 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d800f920 con 0x55d870429cb0 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/746859333 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1e4004050 con 0x55d870429cb0 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1d3400640 1 -- 172.21.15.102:0/746859333 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc1d800fac0 con 0x55d870429cb0 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.624+0000 7fc1d3400640 1 -- 172.21.15.102:0/746859333 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc1d800fcc0 con 0x55d870429cb0 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.625+0000 7fc1f6ffd740 10 client.25217 fetch_fsmap learned FSMap version 27 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.625+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/746859333 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d87043dcf0 con 0x55d870429cb0 2024-04-26T20:34:35.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.625+0000 7fc1d3400640 1 -- 172.21.15.102:0/746859333 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1e4004050 con 0x55d870429cb0 2024-04-26T20:34:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.625+0000 7fc1d3400640 1 -- 172.21.15.102:0/746859333 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc1d801c420 con 0x55d870429cb0 2024-04-26T20:34:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.625+0000 7fc1f6ffd740 10 client.25217 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:34:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.625+0000 7fc1f6ffd740 10 client.25217 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T20:34:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.625+0000 7fc1f6ffd740 20 client.25217 populate_metadata read hostname 'smithi102' 2024-04-26T20:34:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.625+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x55d8704403c0 0x55d870441830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.625+0000 7fc1f6ffd740 4 client.25217 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:34:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.625+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/746859333 --> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d87039b210 con 0x55d8704403c0 2024-04-26T20:34:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.626+0000 7fc1f2600640 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x55d8704403c0 0x55d870441830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.626+0000 7fc1f2600640 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x55d8704403c0 0x55d870441830 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=64114711c4603bb server_cookie=c5397f3aeebd52d9 in_seq=0 out_seq=0 2024-04-26T20:34:35.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.626+0000 7fc1d3400640 10 client.25217 ms_handle_connect on v2:172.21.15.102:6838/3473487681 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1d3400640 1 -- 172.21.15.102:0/746859333 <== mds.1 v2:172.21.15.102:6838/3473487681 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d87039b210 con 0x55d8704403c0 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1d3400640 10 client.25217 handle_command_reply: tid=0 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1f6ffd740 1 client.25217 shutdown 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/746859333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 msgr2=0x55d870433c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d870433c60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec19c0b0 tx=0x7fc1d8012b60 comp rx=0 tx=0).stop 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/746859333 shutdown_connections 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x55d8704403c0 0x55d870441830 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d87043aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1d3400640 0 client.25217 ms_handle_reset on v2:172.21.15.102:6838/3473487681 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d870433c60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/746859333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d870433720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.627+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/746859333 >> 172.21.15.102:0/746859333 conn(0x55d8703bdd10 msgr2=0x55d870423f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.628+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/746859333 shutdown_connections 2024-04-26T20:34:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.628+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/746859333 wait complete. 2024-04-26T20:34:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.628+0000 7fc1f6ffd740 20 client.25217 trim_cache size 0 max 16384 2024-04-26T20:34:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f6ffd740 1 Processor -- start 2024-04-26T20:34:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f6ffd740 1 -- start start 2024-04-26T20:34:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d87041fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d87043ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d870427c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d87043fb70 con 0x55d87042ee20 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d870440240 con 0x55d870429cb0 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d870440910 con 0x55d8704292c0 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f2600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d870427c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f2600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d87041fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f2600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d87041fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55716/0 (socket says 172.21.15.102:55716) 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.648+0000 7fc1f2600640 1 -- 172.21.15.102:0/360360656 learned_addr learned my addr 172.21.15.102:0/360360656 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1f1200640 1 --2- 172.21.15.102:0/360360656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d87043ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1f2600640 1 -- 172.21.15.102:0/360360656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 msgr2=0x55d87043ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1f2600640 1 --2- 172.21.15.102:0/360360656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d87043ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1f2600640 1 -- 172.21.15.102:0/360360656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 msgr2=0x55d870427c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1f2600640 1 --2- 172.21.15.102:0/360360656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d870427c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1f2600640 1 -- 172.21.15.102:0/360360656 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d87042e390 con 0x55d8704292c0 2024-04-26T20:34:35.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1f1200640 1 --2- 172.21.15.102:0/360360656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d87043ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:34:35.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1f2600640 1 --2- 172.21.15.102:0/360360656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d87041fdf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400a010 tx=0x7fc1e4005ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2c54bc582d1b256 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1de000640 1 -- 172.21.15.102:0/360360656 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4005380 con 0x55d8704292c0 2024-04-26T20:34:35.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1de000640 1 -- 172.21.15.102:0/360360656 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1e400f4e0 con 0x55d8704292c0 2024-04-26T20:34:35.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.649+0000 7fc1de000640 1 -- 172.21.15.102:0/360360656 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4028450 con 0x55d8704292c0 2024-04-26T20:34:35.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.650+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/360360656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 msgr2=0x55d87041fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.650+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/360360656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d87041fdf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400a010 tx=0x7fc1e4005ad0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.650+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/360360656 shutdown_connections 2024-04-26T20:34:35.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.650+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/360360656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d87042ee20 0x55d870427c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.650+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/360360656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d870429cb0 0x55d87043ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.650+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/360360656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8704292c0 0x55d87041fdf0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.650+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/360360656 >> 172.21.15.102:0/360360656 conn(0x55d8703bdd10 msgr2=0x55d87042b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.650+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/360360656 shutdown_connections 2024-04-26T20:34:35.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.650+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/360360656 wait complete. 2024-04-26T20:34:35.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.651+0000 7fc1f6ffd740 1 -- start start 2024-04-26T20:34:35.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.651+0000 7fc1f6ffd740 1 Processor -- start 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.651+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8704292c0 0x55d870436e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.651+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d870429cb0 0x55d870437410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.651+0000 7fc1f6ffd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87042ee20 0x55d87043a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.651+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d8704416b0 con 0x55d8704292c0 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.651+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d870440fe0 con 0x55d87042ee20 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.651+0000 7fc1f6ffd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d870441d80 con 0x55d870429cb0 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.651+0000 7fc1f6ffd740 10 client.0 _ll_register_callbacks cb 0x7fc1d8009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f2600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8704292c0 0x55d870436e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f2600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8704292c0 0x55d870436e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38042/0 (socket says 172.21.15.102:38042) 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f1200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d870429cb0 0x55d870437410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f2600640 1 -- 172.21.15.102:0/522231424 learned_addr learned my addr 172.21.15.102:0/522231424 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f2600640 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87042ee20 0x55d87043a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f1200640 1 -- 172.21.15.102:0/522231424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87042ee20 msgr2=0x55d87043a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f1200640 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87042ee20 0x55d87043a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f1200640 1 -- 172.21.15.102:0/522231424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8704292c0 msgr2=0x55d870436e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f1200640 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8704292c0 0x55d870436e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f1200640 1 -- 172.21.15.102:0/522231424 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d87043a6f0 con 0x55d870429cb0 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f2600640 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8704292c0 0x55d870436e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1f1200640 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d870429cb0 0x55d870437410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc1d801c960 tx=0x7fc1d8009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de075a32714b5e55 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.652+0000 7fc1d3400640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1d3400640 1 -- 172.21.15.102:0/522231424 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d801cd60 con 0x55d870429cb0 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/522231424 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1e400f4e0 con 0x55d870429cb0 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1d3400640 1 -- 172.21.15.102:0/522231424 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc1d801c420 con 0x55d870429cb0 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1d3400640 1 -- 172.21.15.102:0/522231424 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc1d801c620 con 0x55d870429cb0 2024-04-26T20:34:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1f6ffd740 10 client.25219 fetch_fsmap learned FSMap version 27 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/522231424 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d8704af340 con 0x55d870429cb0 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1d3400640 1 -- 172.21.15.102:0/522231424 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1e400f4e0 con 0x55d870429cb0 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1d3400640 1 -- 172.21.15.102:0/522231424 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc1d8003a80 con 0x55d870429cb0 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1f6ffd740 10 client.25219 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1f6ffd740 10 client.25219 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1f6ffd740 20 client.25219 populate_metadata read hostname 'smithi102' 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x55d8704b3f00 0x55d8704b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1f6ffd740 4 client.25219 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.653+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/522231424 --> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] -- command(tid 0: {"prefix": "status"}) -- 0x55d870395080 con 0x55d8704b3f00 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.654+0000 7fc1f2600640 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x55d8704b3f00 0x55d8704b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.654+0000 7fc1f2600640 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x55d8704b3f00 0x55d8704b0a10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=59176203b7ebc324 server_cookie=db01ff31d42dcb97 in_seq=0 out_seq=0 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.654+0000 7fc1d3400640 10 client.25219 ms_handle_connect on v2:172.21.15.102:6838/3473487681 2024-04-26T20:34:35.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1d3400640 1 -- 172.21.15.102:0/522231424 <== mds.1 v2:172.21.15.102:6838/3473487681 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d870395080 con 0x55d8704b3f00 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1d3400640 10 client.25219 handle_command_reply: tid=0 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 client.25219 shutdown 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/522231424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d870429cb0 msgr2=0x55d870437410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d870429cb0 0x55d870437410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc1d801c960 tx=0x7fc1d8009310 comp rx=0 tx=0).stop 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/522231424 shutdown_connections 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x55d8704b3f00 0x55d8704b0a10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d87042ee20 0x55d87043a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d870429cb0 0x55d870437410 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 --2- 172.21.15.102:0/522231424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8704292c0 0x55d870436e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1d3400640 0 client.25219 ms_handle_reset on v2:172.21.15.102:6838/3473487681 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/522231424 >> 172.21.15.102:0/522231424 conn(0x55d8703bdd10 msgr2=0x55d8704236d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/522231424 shutdown_connections 2024-04-26T20:34:35.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 1 -- 172.21.15.102:0/522231424 wait complete. 2024-04-26T20:34:35.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.655+0000 7fc1f6ffd740 20 client.25219 trim_cache size 0 max 16384 2024-04-26T20:34:35.662 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:34:35.662 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:34:35.662 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 1, 2024-04-26T20:34:35.662 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24565, 2024-04-26T20:34:35.662 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:34:35.662 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:34:35.662 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:34:35.663 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 104.01377765700001, 2024-04-26T20:34:35.663 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 27, 2024-04-26T20:34:35.663 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-04-26T20:34:35.663 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:34:35.663 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 142.87091469200001 2024-04-26T20:34:35.663 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:34:35.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 -- 172.21.15.102:0/3568637132 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc1d40a3cc0 msgr2=0x7fc1d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc1d40a3cc0 0x7fc1d40a6180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55d87042a720 tx=0x7fc1e00f4af0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 -- 172.21.15.102:0/3568637132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ec101660 msgr2=0x7fc1ec19abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ec101660 0x7fc1ec19abf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0031500 tx=0x7fc1e0003c10 comp rx=0 tx=0).stop 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 -- 172.21.15.102:0/3568637132 shutdown_connections 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc1d40a3cc0 0x7fc1d40a6180 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec103210 0x7fc1ec1a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1ec102850 0x7fc1ec19b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 --2- 172.21.15.102:0/3568637132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ec101660 0x7fc1ec19abf0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 -- 172.21.15.102:0/3568637132 >> 172.21.15.102:0/3568637132 conn(0x7fc1ec0fcbb0 msgr2=0x7fc1ec0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 -- 172.21.15.102:0/3568637132 shutdown_connections 2024-04-26T20:34:35.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.656+0000 7fc1f3000640 1 -- 172.21.15.102:0/3568637132 wait complete. 2024-04-26T20:34:35.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T20:34:35.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.866+0000 7f46e7800640 1 -- 172.21.15.102:0/2999961197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0104b80 msgr2=0x7f46e0104fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.866+0000 7f46e7800640 1 --2- 172.21.15.102:0/2999961197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0104b80 0x7f46e0104fa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f46d4009900 tx=0x7f46d402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.866+0000 7f46e7800640 1 -- 172.21.15.102:0/2999961197 shutdown_connections 2024-04-26T20:34:35.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.866+0000 7f46e7800640 1 --2- 172.21.15.102:0/2999961197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46e0105d70 0x7f46e01061f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.866+0000 7f46e7800640 1 --2- 172.21.15.102:0/2999961197 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46e0106730 0x7f46e010cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.866+0000 7f46e7800640 1 --2- 172.21.15.102:0/2999961197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0104b80 0x7f46e0104fa0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.866+0000 7f46e7800640 1 -- 172.21.15.102:0/2999961197 >> 172.21.15.102:0/2999961197 conn(0x7f46e01000f0 msgr2=0x7f46e0102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.866+0000 7f46e7800640 1 -- 172.21.15.102:0/2999961197 shutdown_connections 2024-04-26T20:34:35.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.866+0000 7f46e7800640 1 -- 172.21.15.102:0/2999961197 wait complete. 2024-04-26T20:34:35.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.867+0000 7f46e7800640 1 Processor -- start 2024-04-26T20:34:35.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46e7800640 1 -- start start 2024-04-26T20:34:35.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46e7800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0104b80 0x7f46e019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46e7800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46e0105d70 0x7f46e019b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46e7800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46e0106730 0x7f46e01a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46e7800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f46e01106d0 con 0x7f46e0105d70 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46e7800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f46e0110550 con 0x7f46e0106730 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46e7800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f46e0110850 con 0x7f46e0104b80 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46e0105d70 0x7f46e019b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46e0105d70 0x7f46e019b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38054/0 (socket says 172.21.15.102:38054) 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46dfe00640 1 -- 172.21.15.102:0/1689940719 learned_addr learned my addr 172.21.15.102:0/1689940719 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.868+0000 7f46e5800640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46e0106730 0x7f46e01a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46e4e00640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0104b80 0x7f46e019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46e5800640 1 -- 172.21.15.102:0/1689940719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0104b80 msgr2=0x7f46e019ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46e5800640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0104b80 0x7f46e019ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46e5800640 1 -- 172.21.15.102:0/1689940719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46e0105d70 msgr2=0x7f46e019b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46e5800640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46e0105d70 0x7f46e019b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46e5800640 1 -- 172.21.15.102:0/1689940719 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e01a2960 con 0x7f46e0106730 2024-04-26T20:34:35.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46e4e00640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0104b80 0x7f46e019ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T20:34:35.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46e5800640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46e0106730 0x7f46e01a22f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009fd0 tx=0x7f46d800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46dd600640 1 -- 172.21.15.102:0/1689940719 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d800ce20 con 0x7f46e0106730 2024-04-26T20:34:35.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46dd600640 1 -- 172.21.15.102:0/1689940719 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46d8005020 con 0x7f46e0106730 2024-04-26T20:34:35.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46e7800640 1 -- 172.21.15.102:0/1689940719 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e01a2c50 con 0x7f46e0106730 2024-04-26T20:34:35.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.869+0000 7f46dd600640 1 -- 172.21.15.102:0/1689940719 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8010690 con 0x7f46e0106730 2024-04-26T20:34:35.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.870+0000 7f46e7800640 1 -- 172.21.15.102:0/1689940719 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e01a3150 con 0x7f46e0106730 2024-04-26T20:34:35.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.873+0000 7f46ea60e740 1 Processor -- start 2024-04-26T20:34:35.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.873+0000 7f46dd600640 1 -- 172.21.15.102:0/1689940719 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f46d8010830 con 0x7f46e0106730 2024-04-26T20:34:35.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.873+0000 7f46ea60e740 1 -- start start 2024-04-26T20:34:35.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 0x55ceb4689510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb4689ae0 0x55ceb468e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 0x55ceb46910e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46ea60e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ceb45ed600 con 0x55ceb468ec50 2024-04-26T20:34:35.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46ea60e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ceb458a100 con 0x55ceb4689ae0 2024-04-26T20:34:35.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46ea60e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ceb45e95c0 con 0x55ceb46890f0 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 0x55ceb4689510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 0x55ceb4689510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55760/0 (socket says 172.21.15.102:55760) 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46dfe00640 1 -- 172.21.15.102:0/2740174979 learned_addr learned my addr 172.21.15.102:0/2740174979 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46dfe00640 1 --2- 172.21.15.102:0/2740174979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 0x55ceb46910e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46e4e00640 1 --2- 172.21.15.102:0/2740174979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb4689ae0 0x55ceb468e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46dfe00640 1 -- 172.21.15.102:0/2740174979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb4689ae0 msgr2=0x55ceb468e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46dfe00640 1 --2- 172.21.15.102:0/2740174979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb4689ae0 0x55ceb468e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46dfe00640 1 -- 172.21.15.102:0/2740174979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 msgr2=0x55ceb46910e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46dfe00640 1 --2- 172.21.15.102:0/2740174979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 0x55ceb46910e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.874+0000 7f46dfe00640 1 -- 172.21.15.102:0/2740174979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb42ff2d0 con 0x55ceb46890f0 2024-04-26T20:34:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.875+0000 7f46dfe00640 1 --2- 172.21.15.102:0/2740174979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 0x55ceb4689510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00aef0 tx=0x7f46cc00a810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c42289325e339820 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.875+0000 7f46d2000640 1 -- 172.21.15.102:0/2740174979 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc005220 con 0x55ceb46890f0 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.875+0000 7f46d2000640 1 -- 172.21.15.102:0/2740174979 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46cc004c50 con 0x55ceb46890f0 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.875+0000 7f46d2000640 1 -- 172.21.15.102:0/2740174979 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc00fd20 con 0x55ceb46890f0 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.875+0000 7f46dd600640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46c80a3c40 0x7f46c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.875+0000 7f46dd600640 1 -- 172.21.15.102:0/1689940719 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f46d80d10a0 con 0x7f46e0106730 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.875+0000 7f46e4e00640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46c80a3c40 0x7f46c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.876+0000 7f46ea60e740 1 -- 172.21.15.102:0/2740174979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 msgr2=0x55ceb4689510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.876+0000 7f46ea60e740 1 --2- 172.21.15.102:0/2740174979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 0x55ceb4689510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00aef0 tx=0x7f46cc00a810 comp rx=0 tx=0).stop 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.876+0000 7f46ea60e740 1 -- 172.21.15.102:0/2740174979 shutdown_connections 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.876+0000 7f46ea60e740 1 --2- 172.21.15.102:0/2740174979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 0x55ceb46910e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.876+0000 7f46ea60e740 1 --2- 172.21.15.102:0/2740174979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb4689ae0 0x55ceb468e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.876+0000 7f46ea60e740 1 --2- 172.21.15.102:0/2740174979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 0x55ceb4689510 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.876+0000 7f46ea60e740 1 -- 172.21.15.102:0/2740174979 >> 172.21.15.102:0/2740174979 conn(0x55ceb461dd10 msgr2=0x55ceb4684b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.876+0000 7f46ea60e740 1 -- 172.21.15.102:0/2740174979 shutdown_connections 2024-04-26T20:34:35.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.876+0000 7f46ea60e740 1 -- 172.21.15.102:0/2740174979 wait complete. 2024-04-26T20:34:35.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.876+0000 7f46e4e00640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46c80a3c40 0x7f46c80a6100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55ceb468a550 tx=0x7f46d4005a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.877+0000 7f46ea60e740 1 -- start start 2024-04-26T20:34:35.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.877+0000 7f46ea60e740 1 Processor -- start 2024-04-26T20:34:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.877+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 0x55ceb46935d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.877+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4693b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.877+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb468ec50 0x55ceb469ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.877+0000 7f46ea60e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ceb4691c00 con 0x55ceb4689ae0 2024-04-26T20:34:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.877+0000 7f46ea60e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ceb4574f70 con 0x55ceb468ec50 2024-04-26T20:34:35.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.877+0000 7f46ea60e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ceb4691d80 con 0x55ceb46890f0 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.877+0000 7f46ea60e740 10 client.0 _ll_register_callbacks cb 0x55ceb45e3480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4693b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46e4e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 0x55ceb46935d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46e5800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb468ec50 0x55ceb469ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4693b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38078/0 (socket says 172.21.15.102:38078) 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46dfe00640 1 -- 172.21.15.102:0/2838995260 learned_addr learned my addr 172.21.15.102:0/2838995260 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46e5800640 1 -- 172.21.15.102:0/2838995260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 msgr2=0x55ceb46935d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46e5800640 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 0x55ceb46935d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46e5800640 1 -- 172.21.15.102:0/2838995260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 msgr2=0x55ceb4693b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46e5800640 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4693b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46e5800640 1 -- 172.21.15.102:0/2838995260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb43c96b0 con 0x55ceb468ec50 2024-04-26T20:34:35.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46dfe00640 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4693b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46e5800640 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb468ec50 0x55ceb469ab50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f46d80ce6a0 tx=0x7f46d809cf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d1dd40c8e240d68 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.878+0000 7f46c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46c6a00640 1 -- 172.21.15.102:0/2838995260 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d80b5020 con 0x55ceb468ec50 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46c6a00640 1 -- 172.21.15.102:0/2838995260 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f46d80b6220 con 0x55ceb468ec50 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46ea60e740 1 -- 172.21.15.102:0/2838995260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46cc004c50 con 0x55ceb468ec50 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46c6a00640 1 -- 172.21.15.102:0/2838995260 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f46d80054d0 con 0x55ceb468ec50 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46ea60e740 10 client.25235 fetch_fsmap learned FSMap version 27 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46ea60e740 1 -- 172.21.15.102:0/2838995260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ceb469dc00 con 0x55ceb468ec50 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46c6a00640 1 -- 172.21.15.102:0/2838995260 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46cc004c50 con 0x55ceb468ec50 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46c6a00640 1 -- 172.21.15.102:0/2838995260 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f46d8010b20 con 0x55ceb468ec50 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46ea60e740 10 client.25235 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46ea60e740 10 client.25235 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:34:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46ea60e740 20 client.25235 populate_metadata read hostname 'smithi102' 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46ea60e740 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x55ceb46a0250 0x55ceb46a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46ea60e740 4 client.25235 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.879+0000 7f46ea60e740 1 -- 172.21.15.102:0/2838995260 --> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ceb45fb210 con 0x55ceb46a0250 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.880+0000 7f46e4e00640 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x55ceb46a0250 0x55ceb46a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.880+0000 7f46c6a00640 10 client.25235 ms_handle_connect on v2:172.21.15.112:6838/2318597180 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.880+0000 7f46e4e00640 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x55ceb46a0250 0x55ceb46a16c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc3f4d05ceb92405 server_cookie=b04efaf6614370cb in_seq=0 out_seq=0 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.881+0000 7f46c6a00640 1 -- 172.21.15.102:0/2838995260 <== mds.0 v2:172.21.15.112:6838/2318597180 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ceb45fb210 con 0x55ceb46a0250 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.881+0000 7f46c6a00640 10 client.25235 handle_command_reply: tid=0 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.881+0000 7f46ea60e740 1 client.25235 shutdown 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 1 -- 172.21.15.102:0/2838995260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb468ec50 msgr2=0x55ceb469ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb468ec50 0x55ceb469ab50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f46d80ce6a0 tx=0x7f46d809cf30 comp rx=0 tx=0).stop 2024-04-26T20:34:35.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 1 -- 172.21.15.102:0/2838995260 shutdown_connections 2024-04-26T20:34:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4693b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x55ceb46a0250 0x55ceb46a16c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb468ec50 0x55ceb469ab50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 1 --2- 172.21.15.102:0/2838995260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb46890f0 0x55ceb46935d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46c6a00640 0 client.25235 ms_handle_reset on v2:172.21.15.112:6838/2318597180 2024-04-26T20:34:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 1 -- 172.21.15.102:0/2838995260 >> 172.21.15.102:0/2838995260 conn(0x55ceb461dd10 msgr2=0x55ceb4683eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 1 -- 172.21.15.102:0/2838995260 shutdown_connections 2024-04-26T20:34:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 1 -- 172.21.15.102:0/2838995260 wait complete. 2024-04-26T20:34:35.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.882+0000 7f46ea60e740 20 client.25235 trim_cache size 0 max 16384 2024-04-26T20:34:35.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.902+0000 7f46ea60e740 1 Processor -- start 2024-04-26T20:34:35.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.902+0000 7f46ea60e740 1 -- start start 2024-04-26T20:34:35.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.902+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb46890f0 0x55ceb46928a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.902+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb4689ae0 0x55ceb469ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.902+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 0x55ceb4692de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.902+0000 7f46ea60e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ceb469fa00 con 0x55ceb468ec50 2024-04-26T20:34:35.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.902+0000 7f46ea60e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ceb46a00d0 con 0x55ceb46890f0 2024-04-26T20:34:35.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.902+0000 7f46ea60e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ceb46a07a0 con 0x55ceb4689ae0 2024-04-26T20:34:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.903+0000 7f46e5800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 0x55ceb4692de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.903+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb4689ae0 0x55ceb469ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.903+0000 7f46e5800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 0x55ceb4692de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38084/0 (socket says 172.21.15.102:38084) 2024-04-26T20:34:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.903+0000 7f46e5800640 1 -- 172.21.15.102:0/1073397500 learned_addr learned my addr 172.21.15.102:0/1073397500 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.903+0000 7f46e4e00640 1 --2- 172.21.15.102:0/1073397500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb46890f0 0x55ceb46928a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.903+0000 7f46e5800640 1 -- 172.21.15.102:0/1073397500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb4689ae0 msgr2=0x55ceb469ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.903+0000 7f46e5800640 1 --2- 172.21.15.102:0/1073397500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb4689ae0 0x55ceb469ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.903+0000 7f46e5800640 1 -- 172.21.15.102:0/1073397500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb46890f0 msgr2=0x55ceb46928a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.903+0000 7f46e5800640 1 --2- 172.21.15.102:0/1073397500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb46890f0 0x55ceb46928a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.903+0000 7f46e5800640 1 -- 172.21.15.102:0/1073397500 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb468e280 con 0x55ceb468ec50 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.904+0000 7f46e5800640 1 --2- 172.21.15.102:0/1073397500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 0x55ceb4692de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f46d809cb70 tx=0x7f46d809cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2961775ad92244 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.904+0000 7f46d1600640 1 -- 172.21.15.102:0/1073397500 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8080070 con 0x55ceb468ec50 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.904+0000 7f46d1600640 1 -- 172.21.15.102:0/1073397500 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46d80a1220 con 0x55ceb468ec50 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.904+0000 7f46d1600640 1 -- 172.21.15.102:0/1073397500 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d80b5070 con 0x55ceb468ec50 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.904+0000 7f46ea60e740 1 -- 172.21.15.102:0/1073397500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 msgr2=0x55ceb4692de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.904+0000 7f46ea60e740 1 --2- 172.21.15.102:0/1073397500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 0x55ceb4692de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f46d809cb70 tx=0x7f46d809cd00 comp rx=0 tx=0).stop 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.905+0000 7f46ea60e740 1 -- 172.21.15.102:0/1073397500 shutdown_connections 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.905+0000 7f46ea60e740 1 --2- 172.21.15.102:0/1073397500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb468ec50 0x55ceb4692de0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.905+0000 7f46ea60e740 1 --2- 172.21.15.102:0/1073397500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb4689ae0 0x55ceb469ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.905+0000 7f46ea60e740 1 --2- 172.21.15.102:0/1073397500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb46890f0 0x55ceb46928a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.905+0000 7f46ea60e740 1 -- 172.21.15.102:0/1073397500 >> 172.21.15.102:0/1073397500 conn(0x55ceb461dd10 msgr2=0x55ceb468afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.905+0000 7f46ea60e740 1 -- 172.21.15.102:0/1073397500 shutdown_connections 2024-04-26T20:34:35.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.905+0000 7f46ea60e740 1 -- 172.21.15.102:0/1073397500 wait complete. 2024-04-26T20:34:35.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.905+0000 7f46ea60e740 1 -- start start 2024-04-26T20:34:35.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.905+0000 7f46ea60e740 1 Processor -- start 2024-04-26T20:34:35.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb46890f0 0x55ceb4696d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4697260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46ea60e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb468ec50 0x55ceb469a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46ea60e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ceb46a1540 con 0x55ceb4689ae0 2024-04-26T20:34:35.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46ea60e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ceb46a0e70 con 0x55ceb46890f0 2024-04-26T20:34:35.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46ea60e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ceb46a1c10 con 0x55ceb468ec50 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46ea60e740 10 client.0 _ll_register_callbacks cb 0x7f46d8099190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4697260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46e4e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb46890f0 0x55ceb4696d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4697260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38090/0 (socket says 172.21.15.102:38090) 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46dfe00640 1 -- 172.21.15.102:0/3271746079 learned_addr learned my addr 172.21.15.102:0/3271746079 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46e5800640 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb468ec50 0x55ceb469a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46dfe00640 1 -- 172.21.15.102:0/3271746079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb468ec50 msgr2=0x55ceb469a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46dfe00640 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb468ec50 0x55ceb469a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46dfe00640 1 -- 172.21.15.102:0/3271746079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb46890f0 msgr2=0x55ceb4696d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46dfe00640 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb46890f0 0x55ceb4696d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.906+0000 7f46dfe00640 1 -- 172.21.15.102:0/3271746079 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb469a870 con 0x55ceb4689ae0 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.907+0000 7f46dfe00640 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4697260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55ceb4694ab0 tx=0x7f46cc005560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7efe6714644b13ef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.907+0000 7f46c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:34:35.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.907+0000 7f46c6a00640 1 -- 172.21.15.102:0/3271746079 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc021040 con 0x55ceb4689ae0 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.907+0000 7f46c6a00640 1 -- 172.21.15.102:0/3271746079 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f46cc00fa30 con 0x55ceb4689ae0 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.907+0000 7f46ea60e740 1 -- 172.21.15.102:0/3271746079 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46d80a1220 con 0x55ceb4689ae0 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.907+0000 7f46c6a00640 1 -- 172.21.15.102:0/3271746079 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f46cc035050 con 0x55ceb4689ae0 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.907+0000 7f46ea60e740 10 client.15114 fetch_fsmap learned FSMap version 27 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.907+0000 7f46ea60e740 1 -- 172.21.15.102:0/3271746079 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ceb470f560 con 0x55ceb4689ae0 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.908+0000 7f46c6a00640 1 -- 172.21.15.102:0/3271746079 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46d80a1220 con 0x55ceb4689ae0 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.908+0000 7f46c6a00640 1 -- 172.21.15.102:0/3271746079 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f46cc01b070 con 0x55ceb4689ae0 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.908+0000 7f46ea60e740 10 client.15114 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.908+0000 7f46ea60e740 10 client.15114 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.908+0000 7f46ea60e740 20 client.15114 populate_metadata read hostname 'smithi102' 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.908+0000 7f46ea60e740 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x55ceb4713f70 0x55ceb4710a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.908+0000 7f46ea60e740 4 client.15114 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.908+0000 7f46ea60e740 1 -- 172.21.15.102:0/3271746079 --> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] -- command(tid 0: {"prefix": "status"}) -- 0x55ceb45f5080 con 0x55ceb4713f70 2024-04-26T20:34:35.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.908+0000 7f46e4e00640 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x55ceb4713f70 0x55ceb4710a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46e4e00640 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x55ceb4713f70 0x55ceb4710a80 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20142d0e98e415e5 server_cookie=e1dcc78c43f20c48 in_seq=0 out_seq=0 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46c6a00640 10 client.15114 ms_handle_connect on v2:172.21.15.112:6838/2318597180 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46c6a00640 1 -- 172.21.15.102:0/3271746079 <== mds.0 v2:172.21.15.112:6838/2318597180 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ceb45f5080 con 0x55ceb4713f70 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46c6a00640 10 client.15114 handle_command_reply: tid=0 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 client.15114 shutdown 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 -- 172.21.15.102:0/3271746079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 msgr2=0x55ceb4697260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4697260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55ceb4694ab0 tx=0x7f46cc005560 comp rx=0 tx=0).stop 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 -- 172.21.15.102:0/3271746079 shutdown_connections 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb4689ae0 0x55ceb4697260 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x55ceb4713f70 0x55ceb4710a80 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ceb468ec50 0x55ceb469a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 --2- 172.21.15.102:0/3271746079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ceb46890f0 0x55ceb4696d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46c6a00640 0 client.15114 ms_handle_reset on v2:172.21.15.112:6838/2318597180 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 -- 172.21.15.102:0/3271746079 >> 172.21.15.102:0/3271746079 conn(0x55ceb461dd10 msgr2=0x55ceb46928a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 -- 172.21.15.102:0/3271746079 shutdown_connections 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 1 -- 172.21.15.102:0/3271746079 wait complete. 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.909+0000 7f46ea60e740 20 client.15114 trim_cache size 0 max 16384 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24593, 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:34:35.916 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:34:35.917 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 129.602580996, 2024-04-26T20:34:35.917 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 27, 2024-04-26T20:34:35.917 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-04-26T20:34:35.917 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:34:35.917 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 135.96031233900001 2024-04-26T20:34:35.917 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 -- 172.21.15.102:0/1689940719 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46c80a3c40 msgr2=0x7f46c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46c80a3c40 0x7f46c80a6100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55ceb468a550 tx=0x7f46d4005a40 comp rx=0 tx=0).stop 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 -- 172.21.15.102:0/1689940719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46e0106730 msgr2=0x7f46e01a22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46e0106730 0x7f46e01a22f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009fd0 tx=0x7f46d800ef10 comp rx=0 tx=0).stop 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 -- 172.21.15.102:0/1689940719 shutdown_connections 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46c80a3c40 0x7f46c80a6100 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46e0105d70 0x7f46e019b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46e0106730 0x7f46e01a22f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 --2- 172.21.15.102:0/1689940719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0104b80 0x7f46e019ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 -- 172.21.15.102:0/1689940719 >> 172.21.15.102:0/1689940719 conn(0x7f46e01000f0 msgr2=0x7f46e0109070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 -- 172.21.15.102:0/1689940719 shutdown_connections 2024-04-26T20:34:35.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:35.910+0000 7f46e7800640 1 -- 172.21.15.102:0/1689940719 wait complete. 2024-04-26T20:34:35.929 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T20:34:35.930 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T20:34:35.940 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-04-26T20:34:35.940 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T20:34:36.398 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:37.724 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:37 smithi102 ceph-mon[28094]: pgmap v387: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s, 1 objects/s recovering 2024-04-26T20:34:38.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:37 smithi112 ceph-mon[32769]: pgmap v387: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s, 1 objects/s recovering 2024-04-26T20:34:38.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:37 smithi130 ceph-mon[32704]: pgmap v387: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s, 1 objects/s recovering 2024-04-26T20:34:38.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.592+0000 7fd24c800700 1 -- 172.21.15.102:0/3616027624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd244075e80 msgr2=0x7fd244076300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:38.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.592+0000 7fd24c800700 1 --2- 172.21.15.102:0/3616027624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd244075e80 0x7fd244076300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd234009990 tx=0x7fd234009ca0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.592+0000 7fd24c800700 1 -- 172.21.15.102:0/3616027624 shutdown_connections 2024-04-26T20:34:38.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.592+0000 7fd24c800700 1 --2- 172.21.15.102:0/3616027624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd24410fa70 0x7fd244111e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.592+0000 7fd24c800700 1 --2- 172.21.15.102:0/3616027624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd244077450 0x7fd2440758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.592+0000 7fd24c800700 1 --2- 172.21.15.102:0/3616027624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd244075e80 0x7fd244076300 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.592+0000 7fd24c800700 1 -- 172.21.15.102:0/3616027624 >> 172.21.15.102:0/3616027624 conn(0x7fd2440fe9f0 msgr2=0x7fd244100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:38.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.592+0000 7fd24c800700 1 -- 172.21.15.102:0/3616027624 shutdown_connections 2024-04-26T20:34:38.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.592+0000 7fd24c800700 1 -- 172.21.15.102:0/3616027624 wait complete. 2024-04-26T20:34:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.593+0000 7fd24c800700 1 Processor -- start 2024-04-26T20:34:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd24c800700 1 -- start start 2024-04-26T20:34:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd24c800700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd244075e80 0x7fd24419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd24c800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd244077450 0x7fd2441a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd24c800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd24410fa70 0x7fd2441a7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd24c800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd244115540 con 0x7fd244077450 2024-04-26T20:34:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd24c800700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2441153c0 con 0x7fd24410fa70 2024-04-26T20:34:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd24c800700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd2441156c0 con 0x7fd244075e80 2024-04-26T20:34:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd249400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd244077450 0x7fd2441a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd249e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd244075e80 0x7fd24419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd24a800700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd24410fa70 0x7fd2441a7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd249e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd244075e80 0x7fd24419fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55826/0 (socket says 172.21.15.102:55826) 2024-04-26T20:34:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.594+0000 7fd249e00700 1 -- 172.21.15.102:0/2883227107 learned_addr learned my addr 172.21.15.102:0/2883227107 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.595+0000 7fd249e00700 1 -- 172.21.15.102:0/2883227107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd24410fa70 msgr2=0x7fd2441a7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.595+0000 7fd249e00700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd24410fa70 0x7fd2441a7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.595+0000 7fd249e00700 1 -- 172.21.15.102:0/2883227107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd244077450 msgr2=0x7fd2441a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.595+0000 7fd249e00700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd244077450 0x7fd2441a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.595+0000 7fd249e00700 1 -- 172.21.15.102:0/2883227107 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2441a7860 con 0x7fd244075e80 2024-04-26T20:34:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.595+0000 7fd249e00700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd244075e80 0x7fd24419fc60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd23800e970 tx=0x7fd23800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.595+0000 7fd23ea00700 1 -- 172.21.15.102:0/2883227107 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238004a30 con 0x7fd244075e80 2024-04-26T20:34:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.596+0000 7fd23ea00700 1 -- 172.21.15.102:0/2883227107 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd23800cc90 con 0x7fd244075e80 2024-04-26T20:34:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.596+0000 7fd24c800700 1 -- 172.21.15.102:0/2883227107 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2441a7b50 con 0x7fd244075e80 2024-04-26T20:34:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.596+0000 7fd23ea00700 1 -- 172.21.15.102:0/2883227107 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238005610 con 0x7fd244075e80 2024-04-26T20:34:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.596+0000 7fd24c800700 1 -- 172.21.15.102:0/2883227107 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2441a80e0 con 0x7fd244075e80 2024-04-26T20:34:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.597+0000 7fd24c800700 1 -- 172.21.15.102:0/2883227107 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd24406a110 con 0x7fd244075e80 2024-04-26T20:34:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.599+0000 7fd23ea00700 1 -- 172.21.15.102:0/2883227107 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fd23800c7e0 con 0x7fd244075e80 2024-04-26T20:34:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.600+0000 7fd23ea00700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd22c0a3d50 0x7fd22c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.600+0000 7fd23ea00700 1 -- 172.21.15.102:0/2883227107 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fd2380d0ca0 con 0x7fd244075e80 2024-04-26T20:34:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.601+0000 7fd249400700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd22c0a3d50 0x7fd22c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.601+0000 7fd249400700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd22c0a3d50 0x7fd22c0a6210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd23400b2e0 tx=0x7fd234005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.603+0000 7fd23ea00700 1 -- 172.21.15.102:0/2883227107 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd23809d110 con 0x7fd244075e80 2024-04-26T20:34:38.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.851+0000 7fd24c800700 1 -- 172.21.15.102:0/2883227107 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fd244063860 con 0x7fd22c0a3d50 2024-04-26T20:34:38.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.855+0000 7fd23ea00700 1 -- 172.21.15.102:0/2883227107 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fd244063860 con 0x7fd22c0a3d50 2024-04-26T20:34:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:Backend: cephadm 2024-04-26T20:34:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:Available: Yes 2024-04-26T20:34:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:Paused: No 2024-04-26T20:34:38.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 -- 172.21.15.102:0/2883227107 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd22c0a3d50 msgr2=0x7fd22c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:38.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd22c0a3d50 0x7fd22c0a6210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd23400b2e0 tx=0x7fd234005fb0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 -- 172.21.15.102:0/2883227107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd244075e80 msgr2=0x7fd24419fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:38.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd244075e80 0x7fd24419fc60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd23800e970 tx=0x7fd23800ec80 comp rx=0 tx=0).stop 2024-04-26T20:34:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 -- 172.21.15.102:0/2883227107 shutdown_connections 2024-04-26T20:34:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd22c0a3d50 0x7fd22c0a6210 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd24410fa70 0x7fd2441a7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd244075e80 0x7fd24419fc60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 --2- 172.21.15.102:0/2883227107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd244077450 0x7fd2441a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 -- 172.21.15.102:0/2883227107 >> 172.21.15.102:0/2883227107 conn(0x7fd2440fe9f0 msgr2=0x7fd244102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.858+0000 7fd24c800700 1 -- 172.21.15.102:0/2883227107 shutdown_connections 2024-04-26T20:34:38.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:38.859+0000 7fd24c800700 1 -- 172.21.15.102:0/2883227107 wait complete. 2024-04-26T20:34:39.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T20:34:39.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:39 smithi102 ceph-mon[28094]: pgmap v388: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 144 B/s, 1 objects/s recovering 2024-04-26T20:34:40.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:39 smithi112 ceph-mon[32769]: pgmap v388: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 144 B/s, 1 objects/s recovering 2024-04-26T20:34:40.033 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:40.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:39 smithi130 ceph-mon[32704]: pgmap v388: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 144 B/s, 1 objects/s recovering 2024-04-26T20:34:40.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:40 smithi102 ceph-mon[28094]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:41.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:40 smithi112 ceph-mon[32769]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:41.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:40 smithi130 ceph-mon[32704]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:41.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:41 smithi102 ceph-mon[28094]: pgmap v389: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 op/s; 128 B/s, 1 objects/s recovering 2024-04-26T20:34:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:41 smithi112 ceph-mon[32769]: pgmap v389: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 op/s; 128 B/s, 1 objects/s recovering 2024-04-26T20:34:42.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:41 smithi130 ceph-mon[32704]: pgmap v389: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 op/s; 128 B/s, 1 objects/s recovering 2024-04-26T20:34:42.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.238+0000 7f46f6000700 1 -- 172.21.15.102:0/2490247746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46f0102620 msgr2=0x7f46f0102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:42.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.238+0000 7f46f6000700 1 --2- 172.21.15.102:0/2490247746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46f0102620 0x7f46f0102aa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f46dc009990 tx=0x7f46dc009ca0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.238+0000 7f46f6000700 1 -- 172.21.15.102:0/2490247746 shutdown_connections 2024-04-26T20:34:42.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.238+0000 7f46f6000700 1 --2- 172.21.15.102:0/2490247746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f0102fe0 0x7f46f010a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.238+0000 7f46f6000700 1 --2- 172.21.15.102:0/2490247746 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f0103a50 0x7f46f0103e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.238+0000 7f46f6000700 1 --2- 172.21.15.102:0/2490247746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46f0102620 0x7f46f0102aa0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.238+0000 7f46f6000700 1 -- 172.21.15.102:0/2490247746 >> 172.21.15.102:0/2490247746 conn(0x7f46f00fd0f0 msgr2=0x7f46f00ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:42.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.238+0000 7f46f6000700 1 -- 172.21.15.102:0/2490247746 shutdown_connections 2024-04-26T20:34:42.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.238+0000 7f46f6000700 1 -- 172.21.15.102:0/2490247746 wait complete. 2024-04-26T20:34:42.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.239+0000 7f46f6000700 1 Processor -- start 2024-04-26T20:34:42.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.239+0000 7f46f6000700 1 -- start start 2024-04-26T20:34:42.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46f6000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f0102620 0x7f46f0199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:42.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46f6000700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f0102fe0 0x7f46f0199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:42.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46f6000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46f0103a50 0x7f46f01a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:42.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46f6000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f46f010deb0 con 0x7f46f0102620 2024-04-26T20:34:42.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46f6000700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f46f010dd30 con 0x7f46f0102fe0 2024-04-26T20:34:42.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46f6000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f46f010e030 con 0x7f46f0103a50 2024-04-26T20:34:42.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46ef400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f0102620 0x7f46f0199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:42.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46ef400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f0102620 0x7f46f0199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38112/0 (socket says 172.21.15.102:38112) 2024-04-26T20:34:42.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46ef400700 1 -- 172.21.15.102:0/3823871449 learned_addr learned my addr 172.21.15.102:0/3823871449 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:42.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46eea00700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f0102fe0 0x7f46f0199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:42.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.240+0000 7f46efe00700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46f0103a50 0x7f46f01a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:42.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.241+0000 7f46eea00700 1 -- 172.21.15.102:0/3823871449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46f0103a50 msgr2=0x7f46f01a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:42.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.241+0000 7f46eea00700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46f0103a50 0x7f46f01a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.241+0000 7f46eea00700 1 -- 172.21.15.102:0/3823871449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f0102620 msgr2=0x7f46f0199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:42.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.241+0000 7f46eea00700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f0102620 0x7f46f0199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.241+0000 7f46eea00700 1 -- 172.21.15.102:0/3823871449 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f01a1270 con 0x7f46f0102fe0 2024-04-26T20:34:42.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.241+0000 7f46ef400700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f0102620 0x7f46f0199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:42.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.241+0000 7f46eea00700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f0102fe0 0x7f46f0199b50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f46dc000c00 tx=0x7f46dc0057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:42.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.241+0000 7f46e7e00700 1 -- 172.21.15.102:0/3823871449 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc026070 con 0x7f46f0102fe0 2024-04-26T20:34:42.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.241+0000 7f46e7e00700 1 -- 172.21.15.102:0/3823871449 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46dc021410 con 0x7f46f0102fe0 2024-04-26T20:34:42.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.242+0000 7f46e7e00700 1 -- 172.21.15.102:0/3823871449 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc017650 con 0x7f46f0102fe0 2024-04-26T20:34:42.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.242+0000 7f46f6000700 1 -- 172.21.15.102:0/3823871449 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f01a1500 con 0x7f46f0102fe0 2024-04-26T20:34:42.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.242+0000 7f46f6000700 1 -- 172.21.15.102:0/3823871449 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f01a1900 con 0x7f46f0102fe0 2024-04-26T20:34:42.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.243+0000 7f46f6000700 1 -- 172.21.15.102:0/3823871449 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f01a1b60 con 0x7f46f0102fe0 2024-04-26T20:34:42.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.245+0000 7f46e7e00700 1 -- 172.21.15.102:0/3823871449 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f46dc029070 con 0x7f46f0102fe0 2024-04-26T20:34:42.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.247+0000 7f46e7e00700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46d40a3d50 0x7f46d40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:42.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.247+0000 7f46e7e00700 1 -- 172.21.15.102:0/3823871449 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f46dc0d1730 con 0x7f46f0102fe0 2024-04-26T20:34:42.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.247+0000 7f46ef400700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46d40a3d50 0x7f46d40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:42.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.248+0000 7f46ef400700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46d40a3d50 0x7f46d40a6210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f46e0011590 tx=0x7f46e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:42.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.251+0000 7f46e7e00700 1 -- 172.21.15.102:0/3823871449 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f46dc05e060 con 0x7f46f0102fe0 2024-04-26T20:34:42.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.490+0000 7f46f6000700 1 -- 172.21.15.102:0/3823871449 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f46f01a1d30 con 0x7f46d40a3d50 2024-04-26T20:34:42.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.506+0000 7f46e7e00700 1 -- 172.21.15.102:0/3823871449 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f46f01a1d30 con 0x7f46d40a3d50 2024-04-26T20:34:42.507 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T20:34:42.507 INFO:teuthology.orchestra.run.smithi102.stdout:mds.a smithi102 running (2m) 27s ago 2m 12.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 1d74e291a0ea 2024-04-26T20:34:42.507 INFO:teuthology.orchestra.run.smithi102.stdout:mds.b smithi112 running (2m) 74s ago 2m 26.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 bce7f06a9fee 2024-04-26T20:34:42.507 INFO:teuthology.orchestra.run.smithi102.stdout:mds.c smithi130 running (2m) 74s ago 2m 18.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 dd597cbb2214 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mds.d smithi102 running (2m) 27s ago 2m 15.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 de3b9911ca8d 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mds.e smithi112 running (2m) 74s ago 2m 18.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 29e7af1817b9 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mds.f smithi130 running (2m) 74s ago 2m 16.8M - 19.0.0-3306-gbc612ffb 55dc54019ad8 d3e425d7e51f 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mds.g smithi102 running (2m) 27s ago 2m 16.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 facab8299a72 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mds.h smithi112 running (2m) 74s ago 2m 17.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 0d63dc159fe0 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mds.i smithi130 running (2m) 74s ago 2m 17.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 140f6230e432 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mds.j smithi102 running (2m) 27s ago 2m 12.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 0badd3389a91 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mds.k smithi112 running (2m) 74s ago 2m 17.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 1867ebd92353 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mds.l smithi130 running (2m) 74s ago 2m 15.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 5e5f8f74aad5 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi102 *:9283,8765 running (11m) 27s ago 11m 516M - 19.0.0-3306-gbc612ffb 55dc54019ad8 20c510044d08 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi112 *:8443,8765 running (8m) 74s ago 8m 454M - 19.0.0-3306-gbc612ffb 55dc54019ad8 95b0502d8a59 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.z smithi130 *:8443,8765 running (8m) 74s ago 8m 454M - 19.0.0-3306-gbc612ffb 55dc54019ad8 92d42ddf2dc5 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (11m) 27s ago 11m 48.5M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 747c74129f19 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi112 running (9m) 74s ago 9m 41.6M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 8b3a8b0d25b7 2024-04-26T20:34:42.508 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi130 running (9m) 74s ago 9m 42.9M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 a8d92a4102ea 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (8m) 27s ago 8m 45.3M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 3a32bdd8e34a 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi112 running (8m) 74s ago 8m 46.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 b79e65a3ab08 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi130 running (7m) 74s ago 7m 46.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 faa050cbbe8f 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (7m) 27s ago 7m 43.4M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 4a80dc5ed5bc 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi112 running (6m) 74s ago 6m 50.3M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 506a2ce608b9 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi130 running (6m) 74s ago 6m 47.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 765151e63811 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi102 running (5m) 27s ago 5m 51.0M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 bd0514a4e41b 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi112 running (5m) 74s ago 5m 49.1M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 b77c7bb63145 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.8 smithi130 running (4m) 74s ago 4m 44.1M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 0099655d5f37 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.9 smithi102 running (4m) 27s ago 4m 42.4M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 ae79d4304d92 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.10 smithi112 running (3m) 74s ago 3m 42.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 138d10015cac 2024-04-26T20:34:42.509 INFO:teuthology.orchestra.run.smithi102.stdout:osd.11 smithi130 running (3m) 74s ago 3m 48.0M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 a60b3e002627 2024-04-26T20:34:42.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 -- 172.21.15.102:0/3823871449 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46d40a3d50 msgr2=0x7f46d40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:42.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46d40a3d50 0x7f46d40a6210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f46e0011590 tx=0x7f46e0009260 comp rx=0 tx=0).stop 2024-04-26T20:34:42.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 -- 172.21.15.102:0/3823871449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f0102fe0 msgr2=0x7f46f0199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:42.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f0102fe0 0x7f46f0199b50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f46dc000c00 tx=0x7f46dc0057a0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 -- 172.21.15.102:0/3823871449 shutdown_connections 2024-04-26T20:34:42.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f46d40a3d50 0x7f46d40a6210 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46f0103a50 0x7f46f01a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f0102620 0x7f46f0199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 --2- 172.21.15.102:0/3823871449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f0102fe0 0x7f46f0199b50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:42.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 -- 172.21.15.102:0/3823871449 >> 172.21.15.102:0/3823871449 conn(0x7f46f00fd0f0 msgr2=0x7f46f00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:42.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 -- 172.21.15.102:0/3823871449 shutdown_connections 2024-04-26T20:34:42.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:42.509+0000 7f46f6000700 1 -- 172.21.15.102:0/3823871449 wait complete. 2024-04-26T20:34:43.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T20:34:43.742 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:43.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:43 smithi102 ceph-mon[28094]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:43.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:43 smithi102 ceph-mon[28094]: pgmap v390: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 op/s; 107 B/s, 1 objects/s recovering 2024-04-26T20:34:43.924 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:43 smithi130 ceph-mon[32704]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:43.924 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:43 smithi130 ceph-mon[32704]: pgmap v390: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 op/s; 107 B/s, 1 objects/s recovering 2024-04-26T20:34:44.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:43 smithi112 ceph-mon[32769]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:44.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:43 smithi112 ceph-mon[32769]: pgmap v390: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 op/s; 107 B/s, 1 objects/s recovering 2024-04-26T20:34:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.900+0000 7f190fe00700 1 -- 172.21.15.102:0/1519085246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1908075e80 msgr2=0x7f1908076300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.900+0000 7f190fe00700 1 --2- 172.21.15.102:0/1519085246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1908075e80 0x7f1908076300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f18f4009990 tx=0x7f18f4009ca0 comp rx=0 tx=0).stop 2024-04-26T20:34:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.900+0000 7f190fe00700 1 -- 172.21.15.102:0/1519085246 shutdown_connections 2024-04-26T20:34:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.900+0000 7f190fe00700 1 --2- 172.21.15.102:0/1519085246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f190810b350 0x7f190810d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.900+0000 7f190fe00700 1 --2- 172.21.15.102:0/1519085246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1908077450 0x7f19080758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.900+0000 7f190fe00700 1 --2- 172.21.15.102:0/1519085246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1908075e80 0x7f1908076300 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.900+0000 7f190fe00700 1 -- 172.21.15.102:0/1519085246 >> 172.21.15.102:0/1519085246 conn(0x7f19080fea20 msgr2=0x7f1908100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.900+0000 7f190fe00700 1 -- 172.21.15.102:0/1519085246 shutdown_connections 2024-04-26T20:34:45.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.900+0000 7f190fe00700 1 -- 172.21.15.102:0/1519085246 wait complete. 2024-04-26T20:34:45.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.901+0000 7f190fe00700 1 Processor -- start 2024-04-26T20:34:45.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.902+0000 7f190fe00700 1 -- start start 2024-04-26T20:34:45.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.902+0000 7f190fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908075e80 0x7f190819b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:45.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.902+0000 7f190fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1908077450 0x7f190819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:45.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.902+0000 7f190fe00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f190810b350 0x7f19081a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:45.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.902+0000 7f190fe00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1908111130 con 0x7f1908077450 2024-04-26T20:34:45.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.902+0000 7f190fe00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1908110fb0 con 0x7f190810b350 2024-04-26T20:34:45.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.902+0000 7f190fe00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f19081112b0 con 0x7f1908075e80 2024-04-26T20:34:45.904 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:45 smithi102 ceph-mon[28094]: pgmap v391: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4 op/s; 106 B/s, 1 objects/s recovering 2024-04-26T20:34:45.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.902+0000 7f190ca00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1908077450 0x7f190819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:45.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.903+0000 7f190de00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f190810b350 0x7f19081a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:45.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.903+0000 7f190de00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f190810b350 0x7f19081a2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:47102/0 (socket says 172.21.15.102:47102) 2024-04-26T20:34:45.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.903+0000 7f190de00700 1 -- 172.21.15.102:0/883628270 learned_addr learned my addr 172.21.15.102:0/883628270 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:45.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.903+0000 7f190d400700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908075e80 0x7f190819b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:45.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.903+0000 7f190ca00700 1 -- 172.21.15.102:0/883628270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908075e80 msgr2=0x7f190819b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:45.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.903+0000 7f190ca00700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908075e80 0x7f190819b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.903+0000 7f190ca00700 1 -- 172.21.15.102:0/883628270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f190810b350 msgr2=0x7f19081a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.903+0000 7f190ca00700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f190810b350 0x7f19081a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.903+0000 7f190ca00700 1 -- 172.21.15.102:0/883628270 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19081a3460 con 0x7f1908077450 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.903+0000 7f190de00700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f190810b350 0x7f19081a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.904+0000 7f190ca00700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1908077450 0x7f190819bd90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1908077180 tx=0x7f18f40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.904+0000 7f1906000700 1 -- 172.21.15.102:0/883628270 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4026070 con 0x7f1908077450 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.904+0000 7f1906000700 1 -- 172.21.15.102:0/883628270 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18f4021410 con 0x7f1908077450 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.904+0000 7f190fe00700 1 -- 172.21.15.102:0/883628270 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19081a36f0 con 0x7f1908077450 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.904+0000 7f190fe00700 1 -- 172.21.15.102:0/883628270 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19081a3b40 con 0x7f1908077450 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.904+0000 7f1906000700 1 -- 172.21.15.102:0/883628270 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4017650 con 0x7f1908077450 2024-04-26T20:34:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.905+0000 7f190fe00700 1 -- 172.21.15.102:0/883628270 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f190806a110 con 0x7f1908077450 2024-04-26T20:34:45.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.907+0000 7f1906000700 1 -- 172.21.15.102:0/883628270 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f18f4029070 con 0x7f1908077450 2024-04-26T20:34:45.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.909+0000 7f1906000700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18f00a3cd0 0x7f18f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:45.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.909+0000 7f1906000700 1 -- 172.21.15.102:0/883628270 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f18f40d15a0 con 0x7f1908077450 2024-04-26T20:34:45.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.909+0000 7f190d400700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18f00a3cd0 0x7f18f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:45.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.910+0000 7f190d400700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18f00a3cd0 0x7f18f00a6190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1908077830 tx=0x7f18fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:45.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:45.914+0000 7f1906000700 1 -- 172.21.15.102:0/883628270 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f18f4013070 con 0x7f1908077450 2024-04-26T20:34:46.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:45 smithi112 ceph-mon[32769]: pgmap v391: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4 op/s; 106 B/s, 1 objects/s recovering 2024-04-26T20:34:46.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:45 smithi130 ceph-mon[32704]: pgmap v391: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4 op/s; 106 B/s, 1 objects/s recovering 2024-04-26T20:34:46.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.148+0000 7f190fe00700 1 -- 172.21.15.102:0/883628270 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f1908063860 con 0x7f18f00a3cd0 2024-04-26T20:34:46.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.164+0000 7f1906000700 1 -- 172.21.15.102:0/883628270 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f1908063860 con 0x7f18f00a3cd0 2024-04-26T20:34:46.165 INFO:teuthology.orchestra.run.smithi102.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T20:34:46.166 INFO:teuthology.orchestra.run.smithi102.stdout:mds.all 12/12 77s ago 3m smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-26T20:34:46.166 INFO:teuthology.orchestra.run.smithi102.stdout:mgr 3/3 77s ago 9m smithi102=x;smithi112=y;smithi130=z;count:3 2024-04-26T20:34:46.166 INFO:teuthology.orchestra.run.smithi102.stdout:mon 3/3 77s ago 9m smithi102:172.21.15.102=a;smithi112:172.21.15.112=b;smithi130:172.21.15.130=c;count:3 2024-04-26T20:34:46.166 INFO:teuthology.orchestra.run.smithi102.stdout:osd 12 77s ago - 2024-04-26T20:34:46.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.167+0000 7f190fe00700 1 -- 172.21.15.102:0/883628270 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18f00a3cd0 msgr2=0x7f18f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:46.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.167+0000 7f190fe00700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18f00a3cd0 0x7f18f00a6190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1908077830 tx=0x7f18fc009260 comp rx=0 tx=0).stop 2024-04-26T20:34:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.167+0000 7f190fe00700 1 -- 172.21.15.102:0/883628270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1908077450 msgr2=0x7f190819bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.167+0000 7f190fe00700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1908077450 0x7f190819bd90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1908077180 tx=0x7f18f40057c0 comp rx=0 tx=0).stop 2024-04-26T20:34:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.168+0000 7f190fe00700 1 -- 172.21.15.102:0/883628270 shutdown_connections 2024-04-26T20:34:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.168+0000 7f190fe00700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f18f00a3cd0 0x7f18f00a6190 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.168+0000 7f190fe00700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f190810b350 0x7f19081a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.168+0000 7f190fe00700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1908075e80 0x7f190819b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:46.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.168+0000 7f190fe00700 1 --2- 172.21.15.102:0/883628270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1908077450 0x7f190819bd90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:46.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.168+0000 7f190fe00700 1 -- 172.21.15.102:0/883628270 >> 172.21.15.102:0/883628270 conn(0x7f19080fea20 msgr2=0x7f1908107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:46.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.168+0000 7f190fe00700 1 -- 172.21.15.102:0/883628270 shutdown_connections 2024-04-26T20:34:46.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:46.168+0000 7f190fe00700 1 -- 172.21.15.102:0/883628270 wait complete. 2024-04-26T20:34:47.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T20:34:47.481 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:47.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:47 smithi102 ceph-mon[28094]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:47.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:47 smithi102 ceph-mon[28094]: pgmap v392: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:34:48.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:47 smithi112 ceph-mon[32769]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:48.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:47 smithi112 ceph-mon[32769]: pgmap v392: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:34:48.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:47 smithi130 ceph-mon[32704]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:48.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:47 smithi130 ceph-mon[32704]: pgmap v392: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:34:49.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.823+0000 7f8889000700 1 -- 172.21.15.102:0/1637373675 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8884109170 msgr2=0x7f888410b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:49.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.823+0000 7f8889000700 1 --2- 172.21.15.102:0/1637373675 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8884109170 0x7f888410b5e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f887c009f90 tx=0x7f887c00b330 comp rx=0 tx=0).stop 2024-04-26T20:34:49.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.823+0000 7f8889000700 1 -- 172.21.15.102:0/1637373675 shutdown_connections 2024-04-26T20:34:49.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.823+0000 7f8889000700 1 --2- 172.21.15.102:0/1637373675 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8884109170 0x7f888410b5e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:49.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.823+0000 7f8889000700 1 --2- 172.21.15.102:0/1637373675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8884103f10 0x7f8884106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:49.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.823+0000 7f8889000700 1 --2- 172.21.15.102:0/1637373675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8884106840 0x7f8884108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:49.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.823+0000 7f8889000700 1 -- 172.21.15.102:0/1637373675 >> 172.21.15.102:0/1637373675 conn(0x7f88840fd920 msgr2=0x7f88840ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:49.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.823+0000 7f8889000700 1 -- 172.21.15.102:0/1637373675 shutdown_connections 2024-04-26T20:34:49.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.823+0000 7f8889000700 1 -- 172.21.15.102:0/1637373675 wait complete. 2024-04-26T20:34:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.825+0000 7f8889000700 1 Processor -- start 2024-04-26T20:34:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.825+0000 7f8889000700 1 -- start start 2024-04-26T20:34:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.825+0000 7f8889000700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8884103f10 0x7f8884199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.825+0000 7f8889000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8884106840 0x7f8884199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.825+0000 7f8889000700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8884109170 0x7f88841a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.825+0000 7f8889000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f888410ef90 con 0x7f8884106840 2024-04-26T20:34:49.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.825+0000 7f8889000700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f888410ee10 con 0x7f8884109170 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.825+0000 7f8889000700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f888410f110 con 0x7f8884103f10 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f8881e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8884103f10 0x7f8884199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f887be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8884106840 0x7f8884199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f8881e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8884103f10 0x7f8884199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:39474/0 (socket says 172.21.15.102:39474) 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f8881e00700 1 -- 172.21.15.102:0/4118789446 learned_addr learned my addr 172.21.15.102:0/4118789446 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f8882800700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8884109170 0x7f88841a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f8881e00700 1 -- 172.21.15.102:0/4118789446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8884109170 msgr2=0x7f88841a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f8881e00700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8884109170 0x7f88841a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f8881e00700 1 -- 172.21.15.102:0/4118789446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8884106840 msgr2=0x7f8884199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f8881e00700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8884106840 0x7f8884199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f8881e00700 1 -- 172.21.15.102:0/4118789446 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88841a1260 con 0x7f8884103f10 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.826+0000 7f8881e00700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8884103f10 0x7f8884199600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f887400eb60 tx=0x7f887400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:49.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.827+0000 7f887aa00700 1 -- 172.21.15.102:0/4118789446 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887400cc90 con 0x7f8884103f10 2024-04-26T20:34:49.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.827+0000 7f887aa00700 1 -- 172.21.15.102:0/4118789446 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88740044d0 con 0x7f8884103f10 2024-04-26T20:34:49.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.827+0000 7f8889000700 1 -- 172.21.15.102:0/4118789446 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88841a1550 con 0x7f8884103f10 2024-04-26T20:34:49.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.827+0000 7f887aa00700 1 -- 172.21.15.102:0/4118789446 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874010430 con 0x7f8884103f10 2024-04-26T20:34:49.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.827+0000 7f8889000700 1 -- 172.21.15.102:0/4118789446 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88841a1a90 con 0x7f8884103f10 2024-04-26T20:34:49.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.828+0000 7f8889000700 1 -- 172.21.15.102:0/4118789446 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f888406a110 con 0x7f8884103f10 2024-04-26T20:34:49.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.830+0000 7f887aa00700 1 -- 172.21.15.102:0/4118789446 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8874003680 con 0x7f8884103f10 2024-04-26T20:34:49.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.831+0000 7f887aa00700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88680a3cd0 0x7f88680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:49.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.831+0000 7f887aa00700 1 -- 172.21.15.102:0/4118789446 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f88740d0b40 con 0x7f8884103f10 2024-04-26T20:34:49.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.831+0000 7f887be00700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88680a3cd0 0x7f88680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:49.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.832+0000 7f887be00700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88680a3cd0 0x7f88680a6190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f886c009750 tx=0x7f886c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:49.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:49.836+0000 7f887aa00700 1 -- 172.21.15.102:0/4118789446 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f887409cfb0 con 0x7f8884103f10 2024-04-26T20:34:50.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:49 smithi112 ceph-mon[32769]: pgmap v393: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:34:50.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.081+0000 7f8889000700 1 -- 172.21.15.102:0/4118789446 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f8884063860 con 0x7f88680a3cd0 2024-04-26T20:34:50.082 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:49 smithi102 ceph-mon[28094]: pgmap v393: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:34:50.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.084+0000 7f887aa00700 1 -- 172.21.15.102:0/4118789446 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f8884063860 con 0x7f88680a3cd0 2024-04-26T20:34:50.084 INFO:teuthology.orchestra.run.smithi102.stdout:HOST ADDR LABELS STATUS 2024-04-26T20:34:50.084 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 172.21.15.102 2024-04-26T20:34:50.084 INFO:teuthology.orchestra.run.smithi102.stdout:smithi112 172.21.15.112 2024-04-26T20:34:50.084 INFO:teuthology.orchestra.run.smithi102.stdout:smithi130 172.21.15.130 2024-04-26T20:34:50.084 INFO:teuthology.orchestra.run.smithi102.stdout:3 hosts in cluster 2024-04-26T20:34:50.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 -- 172.21.15.102:0/4118789446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88680a3cd0 msgr2=0x7f88680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:50.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88680a3cd0 0x7f88680a6190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f886c009750 tx=0x7f886c009310 comp rx=0 tx=0).stop 2024-04-26T20:34:50.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 -- 172.21.15.102:0/4118789446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8884103f10 msgr2=0x7f8884199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:50.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8884103f10 0x7f8884199600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f887400eb60 tx=0x7f887400ee70 comp rx=0 tx=0).stop 2024-04-26T20:34:50.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 -- 172.21.15.102:0/4118789446 shutdown_connections 2024-04-26T20:34:50.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88680a3cd0 0x7f88680a6190 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:50.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8884109170 0x7f88841a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:50.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8884103f10 0x7f8884199600 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:50.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 --2- 172.21.15.102:0/4118789446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8884106840 0x7f8884199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:50.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 -- 172.21.15.102:0/4118789446 >> 172.21.15.102:0/4118789446 conn(0x7f88840fd920 msgr2=0x7f8884104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:50.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 -- 172.21.15.102:0/4118789446 shutdown_connections 2024-04-26T20:34:50.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:50.088+0000 7f8889000700 1 -- 172.21.15.102:0/4118789446 wait complete. 2024-04-26T20:34:50.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:49 smithi130 ceph-mon[32704]: pgmap v393: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:34:50.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T20:34:51.397 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:51 smithi102 ceph-mon[28094]: from='client.25258 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:51.886 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:51 smithi102 ceph-mon[28094]: pgmap v394: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:34:52.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:51 smithi112 ceph-mon[32769]: from='client.25258 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:52.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:51 smithi112 ceph-mon[32769]: pgmap v394: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:34:52.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:51 smithi130 ceph-mon[32704]: from='client.25258 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:52.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:51 smithi130 ceph-mon[32704]: pgmap v394: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:34:53.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.587+0000 7f353b600700 1 -- 172.21.15.102:0/2266009296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3534103580 msgr2=0x7f35341039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:53.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.587+0000 7f353b600700 1 --2- 172.21.15.102:0/2266009296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3534103580 0x7f35341039e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f35200099c0 tx=0x7f3520009cd0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.588+0000 7f353b600700 1 -- 172.21.15.102:0/2266009296 shutdown_connections 2024-04-26T20:34:53.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.588+0000 7f353b600700 1 --2- 172.21.15.102:0/2266009296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534103f20 0x7f353410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.588+0000 7f353b600700 1 --2- 172.21.15.102:0/2266009296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3534102380 0x7f35341027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.588+0000 7f353b600700 1 --2- 172.21.15.102:0/2266009296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3534103580 0x7f35341039e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.588+0000 7f353b600700 1 -- 172.21.15.102:0/2266009296 >> 172.21.15.102:0/2266009296 conn(0x7f35340fd920 msgr2=0x7f35340ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.588+0000 7f353b600700 1 -- 172.21.15.102:0/2266009296 shutdown_connections 2024-04-26T20:34:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.588+0000 7f353b600700 1 -- 172.21.15.102:0/2266009296 wait complete. 2024-04-26T20:34:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.589+0000 7f353b600700 1 Processor -- start 2024-04-26T20:34:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.589+0000 7f353b600700 1 -- start start 2024-04-26T20:34:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.589+0000 7f353b600700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3534102380 0x7f353419b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.589+0000 7f353b600700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534103580 0x7f353419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f353b600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3534103f20 0x7f35341a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f353b600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f353410de50 con 0x7f3534103f20 2024-04-26T20:34:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f353b600700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f353410dcd0 con 0x7f3534103580 2024-04-26T20:34:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f353b600700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f353410dfd0 con 0x7f3534102380 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f3538c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3534102380 0x7f353419b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f3539600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3534103f20 0x7f35341a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f3538c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3534102380 0x7f353419b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:39490/0 (socket says 172.21.15.102:39490) 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f3538c00700 1 -- 172.21.15.102:0/1385056512 learned_addr learned my addr 172.21.15.102:0/1385056512 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f3533e00700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534103580 0x7f353419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f3538c00700 1 -- 172.21.15.102:0/1385056512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534103580 msgr2=0x7f353419bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f3538c00700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534103580 0x7f353419bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f3538c00700 1 -- 172.21.15.102:0/1385056512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3534103f20 msgr2=0x7f35341a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f3538c00700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3534103f20 0x7f35341a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.590+0000 7f3538c00700 1 -- 172.21.15.102:0/1385056512 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35341a3480 con 0x7f3534102380 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.591+0000 7f3538c00700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3534102380 0x7f353419b820 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f352800e9a0 tx=0x7f352800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.591+0000 7f3531600700 1 -- 172.21.15.102:0/1385056512 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528004220 con 0x7f3534102380 2024-04-26T20:34:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.591+0000 7f3531600700 1 -- 172.21.15.102:0/1385056512 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35280043c0 con 0x7f3534102380 2024-04-26T20:34:53.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.591+0000 7f3531600700 1 -- 172.21.15.102:0/1385056512 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528003cc0 con 0x7f3534102380 2024-04-26T20:34:53.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.591+0000 7f353b600700 1 -- 172.21.15.102:0/1385056512 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35341a3770 con 0x7f3534102380 2024-04-26T20:34:53.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.591+0000 7f353b600700 1 -- 172.21.15.102:0/1385056512 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35341a3c60 con 0x7f3534102380 2024-04-26T20:34:53.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.592+0000 7f353b600700 1 -- 172.21.15.102:0/1385056512 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f353406a110 con 0x7f3534102380 2024-04-26T20:34:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.594+0000 7f3531600700 1 -- 172.21.15.102:0/1385056512 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f352800c810 con 0x7f3534102380 2024-04-26T20:34:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.596+0000 7f3531600700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f351c0a3d50 0x7f351c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.596+0000 7f3531600700 1 -- 172.21.15.102:0/1385056512 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f35280d0ba0 con 0x7f3534102380 2024-04-26T20:34:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.596+0000 7f3533e00700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f351c0a3d50 0x7f351c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.597+0000 7f3533e00700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f351c0a3d50 0x7f351c0a6210 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3520003820 tx=0x7f3520005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.598+0000 7f3531600700 1 -- 172.21.15.102:0/1385056512 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f352809d0a0 con 0x7f3534102380 2024-04-26T20:34:53.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.836+0000 7f353b600700 1 -- 172.21.15.102:0/1385056512 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f35340ffb70 con 0x7f351c0a3d50 2024-04-26T20:34:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.839+0000 7f3531600700 1 -- 172.21.15.102:0/1385056512 <== mgr.14152 v2:172.21.15.102:6800/71145250 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f35340ffb70 con 0x7f351c0a3d50 2024-04-26T20:34:53.841 INFO:teuthology.orchestra.run.smithi102.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T20:34:53.841 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VK400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T20:34:53.841 INFO:teuthology.orchestra.run.smithi102.stdout:smithi112 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NV400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-26T20:34:53.841 INFO:teuthology.orchestra.run.smithi102.stdout:smithi130 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-26T20:34:53.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 -- 172.21.15.102:0/1385056512 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f351c0a3d50 msgr2=0x7f351c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:53.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f351c0a3d50 0x7f351c0a6210 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3520003820 tx=0x7f3520005fb0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 -- 172.21.15.102:0/1385056512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3534102380 msgr2=0x7f353419b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:53.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3534102380 0x7f353419b820 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f352800e9a0 tx=0x7f352800ecb0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 -- 172.21.15.102:0/1385056512 shutdown_connections 2024-04-26T20:34:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f351c0a3d50 0x7f351c0a6210 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3534103f20 0x7f35341a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3534102380 0x7f353419b820 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 --2- 172.21.15.102:0/1385056512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3534103580 0x7f353419bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 -- 172.21.15.102:0/1385056512 >> 172.21.15.102:0/1385056512 conn(0x7f35340fd920 msgr2=0x7f35340ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 -- 172.21.15.102:0/1385056512 shutdown_connections 2024-04-26T20:34:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:53.842+0000 7f353b600700 1 -- 172.21.15.102:0/1385056512 wait complete. 2024-04-26T20:34:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:53 smithi112 ceph-mon[32769]: pgmap v395: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:34:54.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:53 smithi130 ceph-mon[32704]: pgmap v395: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:34:54.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:53 smithi102 ceph-mon[28094]: pgmap v395: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:34:54.692 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T20:34:54.701 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-04-26T20:34:54.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T20:34:55.154 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:55.745 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:55 smithi102 ceph-mon[28094]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:55.745 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:55 smithi102 ceph-mon[28094]: pgmap v396: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:34:56.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:55 smithi112 ceph-mon[32769]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:55 smithi112 ceph-mon[32769]: pgmap v396: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:34:56.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:55 smithi130 ceph-mon[32704]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:34:56.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:55 smithi130 ceph-mon[32704]: pgmap v396: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:34:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.375+0000 7f85eba00700 1 -- 172.21.15.102:0/1824473001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85e4102380 msgr2=0x7f85e41027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.375+0000 7f85eba00700 1 --2- 172.21.15.102:0/1824473001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85e4102380 0x7f85e41027a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f85d80099c0 tx=0x7f85d8009cd0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.375+0000 7f85eba00700 1 -- 172.21.15.102:0/1824473001 shutdown_connections 2024-04-26T20:34:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.375+0000 7f85eba00700 1 --2- 172.21.15.102:0/1824473001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85e4103f20 0x7f85e410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.375+0000 7f85eba00700 1 --2- 172.21.15.102:0/1824473001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85e4102380 0x7f85e41027a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.375+0000 7f85eba00700 1 --2- 172.21.15.102:0/1824473001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85e4103580 0x7f85e41039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.375+0000 7f85eba00700 1 -- 172.21.15.102:0/1824473001 >> 172.21.15.102:0/1824473001 conn(0x7f85e40fd920 msgr2=0x7f85e40ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.375+0000 7f85eba00700 1 -- 172.21.15.102:0/1824473001 shutdown_connections 2024-04-26T20:34:57.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.375+0000 7f85eba00700 1 -- 172.21.15.102:0/1824473001 wait complete. 2024-04-26T20:34:57.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.377+0000 7f85eba00700 1 Processor -- start 2024-04-26T20:34:57.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.377+0000 7f85eba00700 1 -- start start 2024-04-26T20:34:57.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85eba00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85e4102380 0x7f85e41062c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:57.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85eba00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85e4103580 0x7f85e4104960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85eba00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85e4103f20 0x7f85e4104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85eba00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f85e410de60 con 0x7f85e4103f20 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85eba00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f85e410dce0 con 0x7f85e4102380 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85eba00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f85e410dfe0 con 0x7f85e4103580 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85e3e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85e4103580 0x7f85e4104960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85e3e00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85e4103580 0x7f85e4104960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55862/0 (socket says 172.21.15.102:55862) 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85e3e00700 1 -- 172.21.15.102:0/3055148429 learned_addr learned my addr 172.21.15.102:0/3055148429 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85e9a00700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85e4103f20 0x7f85e4104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85e9000700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85e4102380 0x7f85e41062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85e3e00700 1 -- 172.21.15.102:0/3055148429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85e4102380 msgr2=0x7f85e41062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85e3e00700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85e4102380 0x7f85e41062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85e3e00700 1 -- 172.21.15.102:0/3055148429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85e4103f20 msgr2=0x7f85e4104ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85e3e00700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85e4103f20 0x7f85e4104ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.378+0000 7f85e3e00700 1 -- 172.21.15.102:0/3055148429 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e4105770 con 0x7f85e4103580 2024-04-26T20:34:57.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.379+0000 7f85e3e00700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85e4103580 0x7f85e4104960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f85d400e9a0 tx=0x7f85d400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:57.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.379+0000 7f85e1600700 1 -- 172.21.15.102:0/3055148429 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d400ca10 con 0x7f85e4103580 2024-04-26T20:34:57.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.379+0000 7f85eba00700 1 -- 172.21.15.102:0/3055148429 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e41a37d0 con 0x7f85e4103580 2024-04-26T20:34:57.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.379+0000 7f85e1600700 1 -- 172.21.15.102:0/3055148429 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85d400cbb0 con 0x7f85e4103580 2024-04-26T20:34:57.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.379+0000 7f85e1600700 1 -- 172.21.15.102:0/3055148429 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d4018680 con 0x7f85e4103580 2024-04-26T20:34:57.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.379+0000 7f85eba00700 1 -- 172.21.15.102:0/3055148429 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e41a3d30 con 0x7f85e4103580 2024-04-26T20:34:57.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.380+0000 7f85eba00700 1 -- 172.21.15.102:0/3055148429 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e406a110 con 0x7f85e4103580 2024-04-26T20:34:57.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.382+0000 7f85e1600700 1 -- 172.21.15.102:0/3055148429 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f85d4010430 con 0x7f85e4103580 2024-04-26T20:34:57.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.384+0000 7f85e1600700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f85cc0a3cd0 0x7f85cc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:34:57.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.384+0000 7f85e1600700 1 -- 172.21.15.102:0/3055148429 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f85d40d0b30 con 0x7f85e4103580 2024-04-26T20:34:57.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.384+0000 7f85e9000700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f85cc0a3cd0 0x7f85cc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:34:57.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.385+0000 7f85e9000700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f85cc0a3cd0 0x7f85cc0a6190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f85d8000c00 tx=0x7f85d8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:34:57.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.388+0000 7f85e1600700 1 -- 172.21.15.102:0/3055148429 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f85d409cfa0 con 0x7f85e4103580 2024-04-26T20:34:57.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.659+0000 7f85eba00700 1 -- 172.21.15.102:0/3055148429 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f85e41a41b0 con 0x7f85e4103580 2024-04-26T20:34:57.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.659+0000 7f85e1600700 1 -- 172.21.15.102:0/3055148429 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3106 (secure 0 0 0) 0x7f85d409c8a0 con 0x7f85e4103580 2024-04-26T20:34:57.661 INFO:teuthology.orchestra.run.smithi102.stdout:e27 2024-04-26T20:34:57.661 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T20:34:57.661 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T20:34:57.661 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-04-26T20:34:57.661 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:57.661 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-04-26T20:34:57.661 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 27 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-04-26T20:32:25.769708+0000 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-04-26T20:32:55.699799+0000 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 300 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 0 2024-04-26T20:34:57.662 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 5 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1,2,3,4 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=24593,1=24565,2=24547,3=24533,4=24526} 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3,4] 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-04-26T20:34:57.663 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 0 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 24593 members: 24565,24593,24547,24533,24526 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b{0:24593} state up:active seq 3 addr [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.f{0:24514} state up:standby-replay seq 1 addr [v2:172.21.15.130:6834/829567775,v1:172.21.15.130:6835/829567775] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.g{1:24565} state up:active seq 11 addr [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.e{1:24506} state up:standby-replay seq 1 addr [v2:172.21.15.112:6834/2010209065,v1:172.21.15.112:6835/2010209065] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.i{2:24547} state up:active seq 12 addr [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.d{2:24493} state up:standby-replay seq 1 addr [v2:172.21.15.102:6836/3283929332,v1:172.21.15.102:6837/3283929332] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.h{3:24533} state up:active seq 13 addr [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.k{3:24458} state up:standby-replay seq 1 addr [v2:172.21.15.112:6832/1543441778,v1:172.21.15.112:6833/1543441778] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c{4:24526} state up:active seq 14 addr [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.l{4:24448} state up:standby-replay seq 1 addr [v2:172.21.15.130:6832/4251888549,v1:172.21.15.130:6833/4251888549] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:57.664 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:57.665 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-04-26T20:34:57.665 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:34:57.665 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.j{-1:14382} state up:standby seq 1 addr [v2:172.21.15.102:6834/3853049115,v1:172.21.15.102:6835/3853049115] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.665 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a{-1:14463} state up:standby seq 1 addr [v2:172.21.15.102:6840/2992258222,v1:172.21.15.102:6841/2992258222] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T20:34:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.662+0000 7f85eba00700 1 -- 172.21.15.102:0/3055148429 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f85cc0a3cd0 msgr2=0x7f85cc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.662+0000 7f85eba00700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f85cc0a3cd0 0x7f85cc0a6190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f85d8000c00 tx=0x7f85d8005fb0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.662+0000 7f85eba00700 1 -- 172.21.15.102:0/3055148429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85e4103580 msgr2=0x7f85e4104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:34:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.662+0000 7f85eba00700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85e4103580 0x7f85e4104960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f85d400e9a0 tx=0x7f85d400ecb0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.662+0000 7f85eba00700 1 -- 172.21.15.102:0/3055148429 shutdown_connections 2024-04-26T20:34:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.662+0000 7f85eba00700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f85cc0a3cd0 0x7f85cc0a6190 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.662+0000 7f85eba00700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85e4103f20 0x7f85e4104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.662+0000 7f85eba00700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85e4102380 0x7f85e41062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.662+0000 7f85eba00700 1 --2- 172.21.15.102:0/3055148429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85e4103580 0x7f85e4104960 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:34:57.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.662+0000 7f85eba00700 1 -- 172.21.15.102:0/3055148429 >> 172.21.15.102:0/3055148429 conn(0x7f85e40fd920 msgr2=0x7f85e406b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:34:57.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.663+0000 7f85eba00700 1 -- 172.21.15.102:0/3055148429 shutdown_connections 2024-04-26T20:34:57.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:34:57.663+0000 7f85eba00700 1 -- 172.21.15.102:0/3055148429 wait complete. 2024-04-26T20:34:57.667 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-26T20:34:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:57 smithi112 ceph-mon[32769]: pgmap v397: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:34:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:57 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3055148429' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T20:34:58.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:57 smithi102 ceph-mon[28094]: pgmap v397: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:34:58.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:57 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3055148429' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T20:34:58.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:57 smithi130 ceph-mon[32704]: pgmap v397: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:34:58.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:57 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3055148429' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T20:34:58.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T20:34:58.974 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:34:59.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:34:59 smithi102 ceph-mon[28094]: pgmap v398: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:35:00.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:34:59 smithi112 ceph-mon[32769]: pgmap v398: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:35:00.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:34:59 smithi130 ceph-mon[32704]: pgmap v398: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:35:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.024+0000 7fa54fe00700 1 -- 172.21.15.102:0/2593833357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5501045b0 msgr2=0x7fa550104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.024+0000 7fa54fe00700 1 --2- 172.21.15.102:0/2593833357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5501045b0 0x7fa550104a10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa538009990 tx=0x7fa538009ca0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.024+0000 7fa54fe00700 1 -- 172.21.15.102:0/2593833357 shutdown_connections 2024-04-26T20:35:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.024+0000 7fa54fe00700 1 --2- 172.21.15.102:0/2593833357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa550104f50 0x7fa55010b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.024+0000 7fa54fe00700 1 --2- 172.21.15.102:0/2593833357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5501033b0 0x7fa5501037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.024+0000 7fa54fe00700 1 --2- 172.21.15.102:0/2593833357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5501045b0 0x7fa550104a10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.024+0000 7fa54fe00700 1 -- 172.21.15.102:0/2593833357 >> 172.21.15.102:0/2593833357 conn(0x7fa5500fe950 msgr2=0x7fa550100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.024+0000 7fa54fe00700 1 -- 172.21.15.102:0/2593833357 shutdown_connections 2024-04-26T20:35:01.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.024+0000 7fa54fe00700 1 -- 172.21.15.102:0/2593833357 wait complete. 2024-04-26T20:35:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.026+0000 7fa54fe00700 1 Processor -- start 2024-04-26T20:35:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.026+0000 7fa54fe00700 1 -- start start 2024-04-26T20:35:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.026+0000 7fa54fe00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5501033b0 0x7fa55019b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.026+0000 7fa54fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5501045b0 0x7fa55019bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.026+0000 7fa54fe00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa550104f50 0x7fa5501a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.026+0000 7fa54fe00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa55010ee40 con 0x7fa5501045b0 2024-04-26T20:35:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.026+0000 7fa54fe00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa55010ecc0 con 0x7fa550104f50 2024-04-26T20:35:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.026+0000 7fa54fe00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa55010efc0 con 0x7fa5501033b0 2024-04-26T20:35:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.027+0000 7fa54ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5501033b0 0x7fa55019b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.027+0000 7fa54ea00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5501033b0 0x7fa55019b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:55880/0 (socket says 172.21.15.102:55880) 2024-04-26T20:35:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.027+0000 7fa54ea00700 1 -- 172.21.15.102:0/1986238390 learned_addr learned my addr 172.21.15.102:0/1986238390 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.027+0000 7fa547e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5501045b0 0x7fa55019bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.027+0000 7fa54f400700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa550104f50 0x7fa5501a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.027+0000 7fa54ea00700 1 -- 172.21.15.102:0/1986238390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa550104f50 msgr2=0x7fa5501a2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.027+0000 7fa54ea00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa550104f50 0x7fa5501a2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.027+0000 7fa54ea00700 1 -- 172.21.15.102:0/1986238390 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5501045b0 msgr2=0x7fa55019bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.027+0000 7fa54ea00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5501045b0 0x7fa55019bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.027+0000 7fa54ea00700 1 -- 172.21.15.102:0/1986238390 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5501a3310 con 0x7fa5501033b0 2024-04-26T20:35:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.028+0000 7fa54ea00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5501033b0 0x7fa55019b700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa54000ea80 tx=0x7fa54000ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.028+0000 7fa547400700 1 -- 172.21.15.102:0/1986238390 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54000cbd0 con 0x7fa5501033b0 2024-04-26T20:35:01.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.028+0000 7fa547400700 1 -- 172.21.15.102:0/1986238390 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa54000cd70 con 0x7fa5501033b0 2024-04-26T20:35:01.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.028+0000 7fa54fe00700 1 -- 172.21.15.102:0/1986238390 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5501a3600 con 0x7fa5501033b0 2024-04-26T20:35:01.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.028+0000 7fa547400700 1 -- 172.21.15.102:0/1986238390 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540018710 con 0x7fa5501033b0 2024-04-26T20:35:01.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.028+0000 7fa54fe00700 1 -- 172.21.15.102:0/1986238390 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5501a3b90 con 0x7fa5501033b0 2024-04-26T20:35:01.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.029+0000 7fa54fe00700 1 -- 172.21.15.102:0/1986238390 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa55006a110 con 0x7fa5501033b0 2024-04-26T20:35:01.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.031+0000 7fa547400700 1 -- 172.21.15.102:0/1986238390 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa54001f020 con 0x7fa5501033b0 2024-04-26T20:35:01.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.032+0000 7fa547400700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5340a3d50 0x7fa5340a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:01.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.032+0000 7fa547400700 1 -- 172.21.15.102:0/1986238390 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fa5400d0a60 con 0x7fa5501033b0 2024-04-26T20:35:01.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.033+0000 7fa547e00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5340a3d50 0x7fa5340a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:01.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.033+0000 7fa547e00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5340a3d50 0x7fa5340a6210 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa53800b720 tx=0x7fa538005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:01.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.035+0000 7fa547400700 1 -- 172.21.15.102:0/1986238390 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa54009ced0 con 0x7fa5501033b0 2024-04-26T20:35:01.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.266+0000 7fa54fe00700 1 -- 172.21.15.102:0/1986238390 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fa5501a3fb0 con 0x7fa5501033b0 2024-04-26T20:35:01.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.267+0000 7fa547400700 1 -- 172.21.15.102:0/1986238390 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5701 (secure 0 0 0) 0x7fa54009c7d0 con 0x7fa5501033b0 2024-04-26T20:35:01.269 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 93 2024-04-26T20:35:01.269 INFO:teuthology.orchestra.run.smithi102.stdout:fsid a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:35:01.269 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-04-26T20:23:20.101386+0000 2024-04-26T20:35:01.269 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-04-26T20:34:30.697401+0000 2024-04-26T20:35:01.269 INFO:teuthology.orchestra.run.smithi102.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T20:35:01.269 INFO:teuthology.orchestra.run.smithi102.stdout:crush_version 27 2024-04-26T20:35:01.269 INFO:teuthology.orchestra.run.smithi102.stdout:full_ratio 0.95 2024-04-26T20:35:01.269 INFO:teuthology.orchestra.run.smithi102.stdout:backfillfull_ratio 0.9 2024-04-26T20:35:01.269 INFO:teuthology.orchestra.run.smithi102.stdout:nearfull_ratio 0.85 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:require_min_compat_client luminous 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:min_compat_client luminous 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:require_osd_release squid 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:stretch_mode_enabled false 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:max_osd 12 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6802/1690985624,v1:172.21.15.102:6803/1690985624] [v2:172.21.15.102:6804/1690985624,v1:172.21.15.102:6805/1690985624] exists,up b00e64c1-d8b2-4ede-a6e5-236cac247fd2 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6800/3112086050,v1:172.21.15.112:6801/3112086050] [v2:172.21.15.112:6802/3112086050,v1:172.21.15.112:6803/3112086050] exists,up e10d054b-2732-48ef-a2ba-4137acda22bb 2024-04-26T20:35:01.270 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6800/2227083221,v1:172.21.15.130:6801/2227083221] [v2:172.21.15.130:6802/2227083221,v1:172.21.15.130:6803/2227083221] exists,up 7d090636-b81e-404e-9b1d-d1919ab42ba3 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6810/819862253,v1:172.21.15.102:6811/819862253] [v2:172.21.15.102:6812/819862253,v1:172.21.15.102:6813/819862253] exists,up a6d4daaf-ac9a-4897-a74a-9327aef16d22 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6808/926818023,v1:172.21.15.112:6809/926818023] [v2:172.21.15.112:6810/926818023,v1:172.21.15.112:6811/926818023] exists,up 4fe3b1fe-1d6f-4ce9-b162-fca16bacd628 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6808/1600420377,v1:172.21.15.130:6809/1600420377] [v2:172.21.15.130:6810/1600420377,v1:172.21.15.130:6811/1600420377] exists,up 7d936c15-c743-4cef-95fc-f394bf75cdf1 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6818/879532763,v1:172.21.15.102:6819/879532763] [v2:172.21.15.102:6820/879532763,v1:172.21.15.102:6821/879532763] exists,up 3bbd0cd9-f5a4-4453-860d-2884ccfa71f6 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 up in weight 1 up_from 51 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6816/971991599,v1:172.21.15.112:6817/971991599] [v2:172.21.15.112:6818/971991599,v1:172.21.15.112:6819/971991599] exists,up 4ebc098f-21d8-4b10-a1cb-2793d0675c44 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6816/3061190043,v1:172.21.15.130:6817/3061190043] [v2:172.21.15.130:6818/3061190043,v1:172.21.15.130:6819/3061190043] exists,up 8a61ad14-24c4-4658-938f-93e58b88ad8a 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6826/4267695892,v1:172.21.15.102:6827/4267695892] [v2:172.21.15.102:6828/4267695892,v1:172.21.15.102:6829/4267695892] exists,up 39e80aea-2edc-49c1-8ab8-46d301e58694 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6824/2204137717,v1:172.21.15.112:6825/2204137717] [v2:172.21.15.112:6826/2204137717,v1:172.21.15.112:6827/2204137717] exists,up bd9ac886-cff6-421d-8d22-fb1680e44c02 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6824/822414281,v1:172.21.15.130:6825/822414281] [v2:172.21.15.130:6826/822414281,v1:172.21.15.130:6827/822414281] exists,up bc84bfd4-0926-4025-bb47-9a6a1e648b32 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 2.a [2,11] 2024-04-26T20:35:01.271 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 2.c [2,1] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 2.10 [8,11] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 2.1b [2,4] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 3.17 [4,8] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 3.2c [4,9] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 3.33 [5,9] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 3.38 [3,9] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 3.3a [4,6] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 4.6 [5,8] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 4.13 [9,10] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 4.1b [9,8] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 4.1c [9,8] 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2611652511 expires 2024-04-27T20:24:28.376337+0000 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6801/1441707916 expires 2024-04-27T20:24:28.376337+0000 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6800/1441707916 expires 2024-04-27T20:24:28.376337+0000 2024-04-26T20:35:01.272 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2809626135 expires 2024-04-27T20:24:28.376337+0000 2024-04-26T20:35:01.273 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2372720525 expires 2024-04-27T20:23:53.695696+0000 2024-04-26T20:35:01.273 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2628958579 expires 2024-04-27T20:23:53.695696+0000 2024-04-26T20:35:01.273 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6800/3976810658 expires 2024-04-27T20:23:53.695696+0000 2024-04-26T20:35:01.273 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6801/3976810658 expires 2024-04-27T20:23:53.695696+0000 2024-04-26T20:35:01.273 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/3271239188 expires 2024-04-27T20:24:28.376337+0000 2024-04-26T20:35:01.273 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/1384283564 expires 2024-04-27T20:23:53.695696+0000 2024-04-26T20:35:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.270+0000 7fa54fe00700 1 -- 172.21.15.102:0/1986238390 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5340a3d50 msgr2=0x7fa5340a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.270+0000 7fa54fe00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5340a3d50 0x7fa5340a6210 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa53800b720 tx=0x7fa538005fd0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.270+0000 7fa54fe00700 1 -- 172.21.15.102:0/1986238390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5501033b0 msgr2=0x7fa55019b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.270+0000 7fa54fe00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5501033b0 0x7fa55019b700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa54000ea80 tx=0x7fa54000ed90 comp rx=0 tx=0).stop 2024-04-26T20:35:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.270+0000 7fa54fe00700 1 -- 172.21.15.102:0/1986238390 shutdown_connections 2024-04-26T20:35:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.270+0000 7fa54fe00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa5340a3d50 0x7fa5340a6210 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.270+0000 7fa54fe00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa550104f50 0x7fa5501a2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.270+0000 7fa54fe00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5501033b0 0x7fa55019b700 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.270+0000 7fa54fe00700 1 --2- 172.21.15.102:0/1986238390 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5501045b0 0x7fa55019bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.270+0000 7fa54fe00700 1 -- 172.21.15.102:0/1986238390 >> 172.21.15.102:0/1986238390 conn(0x7fa5500fe950 msgr2=0x7fa550075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:01.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.271+0000 7fa54fe00700 1 -- 172.21.15.102:0/1986238390 shutdown_connections 2024-04-26T20:35:01.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:01.271+0000 7fa54fe00700 1 -- 172.21.15.102:0/1986238390 wait complete. 2024-04-26T20:35:02.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:01 smithi112 ceph-mon[32769]: pgmap v399: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:35:02.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:01 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1986238390' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T20:35:02.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:01 smithi130 ceph-mon[32704]: pgmap v399: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:35:02.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:01 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1986238390' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T20:35:02.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:01 smithi102 ceph-mon[28094]: pgmap v399: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:35:02.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:01 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1986238390' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T20:35:02.261 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T20:35:02.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:35:02.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.487+0000 7efc37200640 1 -- 172.21.15.102:0/2248710360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc3810a940 msgr2=0x7efc381111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:02.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.487+0000 7efc37200640 1 --2- 172.21.15.102:0/2248710360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc3810a940 0x7efc381111d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efc30009ca0 tx=0x7efc3002a8c0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.487+0000 7efc37200640 1 -- 172.21.15.102:0/2248710360 shutdown_connections 2024-04-26T20:35:02.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.487+0000 7efc37200640 1 --2- 172.21.15.102:0/2248710360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc3810a940 0x7efc381111d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.487+0000 7efc37200640 1 --2- 172.21.15.102:0/2248710360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc38109f80 0x7efc3810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.487+0000 7efc37200640 1 --2- 172.21.15.102:0/2248710360 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc38108d90 0x7efc381091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.487+0000 7efc37200640 1 -- 172.21.15.102:0/2248710360 >> 172.21.15.102:0/2248710360 conn(0x7efc38077d30 msgr2=0x7efc3807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:02.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.487+0000 7efc37200640 1 -- 172.21.15.102:0/2248710360 shutdown_connections 2024-04-26T20:35:02.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.487+0000 7efc37200640 1 -- 172.21.15.102:0/2248710360 wait complete. 2024-04-26T20:35:02.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.488+0000 7efc37200640 1 Processor -- start 2024-04-26T20:35:02.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.488+0000 7efc37200640 1 -- start start 2024-04-26T20:35:02.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.488+0000 7efc37200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc38108d90 0x7efc3819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.488+0000 7efc37200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc38109f80 0x7efc3819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc37200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc3810a940 0x7efc381a6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc37200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efc38114910 con 0x7efc38108d90 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc37200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efc38114790 con 0x7efc38109f80 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc37200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7efc38114a90 con 0x7efc3810a940 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc35e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc38108d90 0x7efc3819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc35e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc38108d90 0x7efc3819ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52908/0 (socket says 172.21.15.102:52908) 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc35e00640 1 -- 172.21.15.102:0/1722006810 learned_addr learned my addr 172.21.15.102:0/1722006810 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc36800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc3810a940 0x7efc381a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc35400640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc38109f80 0x7efc3819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc35e00640 1 -- 172.21.15.102:0/1722006810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc3810a940 msgr2=0x7efc381a6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc35e00640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc3810a940 0x7efc381a6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc35e00640 1 -- 172.21.15.102:0/1722006810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc38109f80 msgr2=0x7efc3819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:02.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc35e00640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc38109f80 0x7efc3819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.489+0000 7efc35e00640 1 -- 172.21.15.102:0/1722006810 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc381a6b70 con 0x7efc38108d90 2024-04-26T20:35:02.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.490+0000 7efc35e00640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc38108d90 0x7efc3819ef50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc2800e9c0 tx=0x7efc2800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:02.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.490+0000 7efc2ea00640 1 -- 172.21.15.102:0/1722006810 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2800cd80 con 0x7efc38108d90 2024-04-26T20:35:02.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.490+0000 7efc2ea00640 1 -- 172.21.15.102:0/1722006810 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc280035f0 con 0x7efc38108d90 2024-04-26T20:35:02.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.490+0000 7efc37200640 1 -- 172.21.15.102:0/1722006810 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc381a6e60 con 0x7efc38108d90 2024-04-26T20:35:02.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.490+0000 7efc2ea00640 1 -- 172.21.15.102:0/1722006810 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc28010690 con 0x7efc38108d90 2024-04-26T20:35:02.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.490+0000 7efc37200640 1 -- 172.21.15.102:0/1722006810 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc381a72c0 con 0x7efc38108d90 2024-04-26T20:35:02.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.491+0000 7efc37200640 1 -- 172.21.15.102:0/1722006810 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc381100b0 con 0x7efc38108d90 2024-04-26T20:35:02.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.493+0000 7efc2ea00640 1 -- 172.21.15.102:0/1722006810 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7efc280037a0 con 0x7efc38108d90 2024-04-26T20:35:02.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.494+0000 7efc2ea00640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efc1c0a3c40 0x7efc1c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:02.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.495+0000 7efc2ea00640 1 -- 172.21.15.102:0/1722006810 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7efc280d0d00 con 0x7efc38108d90 2024-04-26T20:35:02.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.495+0000 7efc35400640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efc1c0a3c40 0x7efc1c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:02.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.495+0000 7efc35400640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efc1c0a3c40 0x7efc1c0a6100 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efc381a0460 tx=0x7efc24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.499+0000 7efc2ea00640 1 -- 172.21.15.102:0/1722006810 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efc2809d170 con 0x7efc38108d90 2024-04-26T20:35:02.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.707+0000 7efc37200640 1 -- 172.21.15.102:0/1722006810 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc3802dd60 con 0x7efc38108d90 2024-04-26T20:35:02.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.709+0000 7efc2ea00640 1 -- 172.21.15.102:0/1722006810 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24833 (secure 0 0 0) 0x7efc2809ca70 con 0x7efc38108d90 2024-04-26T20:35:02.711 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:35:02.711 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:34:30.697401+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:35:02.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 -- 172.21.15.102:0/1722006810 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efc1c0a3c40 msgr2=0x7efc1c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:02.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efc1c0a3c40 0x7efc1c0a6100 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efc381a0460 tx=0x7efc24005f90 comp rx=0 tx=0).stop 2024-04-26T20:35:02.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 -- 172.21.15.102:0/1722006810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc38108d90 msgr2=0x7efc3819ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc38108d90 0x7efc3819ef50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc2800e9c0 tx=0x7efc2800ee80 comp rx=0 tx=0).stop 2024-04-26T20:35:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 -- 172.21.15.102:0/1722006810 shutdown_connections 2024-04-26T20:35:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7efc1c0a3c40 0x7efc1c0a6100 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc3810a940 0x7efc381a6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc38109f80 0x7efc3819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 --2- 172.21.15.102:0/1722006810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc38108d90 0x7efc3819ef50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 -- 172.21.15.102:0/1722006810 >> 172.21.15.102:0/1722006810 conn(0x7efc38077d30 msgr2=0x7efc3810d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 -- 172.21.15.102:0/1722006810 shutdown_connections 2024-04-26T20:35:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.712+0000 7efc37200640 1 -- 172.21.15.102:0/1722006810 wait complete. 2024-04-26T20:35:02.733 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:27:10.527933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:32:23.439237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:32:24.702263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:32:28.348785+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:35:02.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:35:02.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.933+0000 7f0fb9400640 1 -- 172.21.15.102:0/1920522740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb4103b70 msgr2=0x7f0fb4103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:02.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.933+0000 7f0fb9400640 1 --2- 172.21.15.102:0/1920522740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb4103b70 0x7f0fb4103ff0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c009900 tx=0x7f0f9c02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.933+0000 7f0fb9400640 1 -- 172.21.15.102:0/1920522740 shutdown_connections 2024-04-26T20:35:02.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.933+0000 7f0fb9400640 1 --2- 172.21.15.102:0/1920522740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb4103b70 0x7f0fb4103ff0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.933+0000 7f0fb9400640 1 --2- 172.21.15.102:0/1920522740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb4104530 0x7f0fb410adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.933+0000 7f0fb9400640 1 --2- 172.21.15.102:0/1920522740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fb4102980 0x7f0fb4102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.933+0000 7f0fb9400640 1 -- 172.21.15.102:0/1920522740 >> 172.21.15.102:0/1920522740 conn(0x7f0fb40fdf10 msgr2=0x7f0fb4100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:02.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.933+0000 7f0fb9400640 1 -- 172.21.15.102:0/1920522740 shutdown_connections 2024-04-26T20:35:02.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.933+0000 7f0fb9400640 1 -- 172.21.15.102:0/1920522740 wait complete. 2024-04-26T20:35:02.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.934+0000 7f0fb9400640 1 Processor -- start 2024-04-26T20:35:02.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.934+0000 7f0fb9400640 1 -- start start 2024-04-26T20:35:02.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb9400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fb4102980 0x7f0fb419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:02.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb9400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb4103b70 0x7f0fb419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb4104530 0x7f0fb41a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb9400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb410e500 con 0x7f0fb4104530 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb9400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb410e380 con 0x7f0fb4102980 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb9400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb410e680 con 0x7f0fb4103b70 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb2800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fb4102980 0x7f0fb419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb4104530 0x7f0fb41a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb2800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fb4102980 0x7f0fb419ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:44690/0 (socket says 172.21.15.102:44690) 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fabe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb4103b70 0x7f0fb419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb2800640 1 -- 172.21.15.102:0/1538908535 learned_addr learned my addr 172.21.15.102:0/1538908535 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb4104530 0x7f0fb41a2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52932/0 (socket says 172.21.15.102:52932) 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb3200640 1 -- 172.21.15.102:0/1538908535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb4103b70 msgr2=0x7f0fb419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb3200640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb4103b70 0x7f0fb419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb3200640 1 -- 172.21.15.102:0/1538908535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fb4102980 msgr2=0x7f0fb419ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:02.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb3200640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fb4102980 0x7f0fb419ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:02.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.935+0000 7f0fb3200640 1 -- 172.21.15.102:0/1538908535 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb41a2a00 con 0x7f0fb4104530 2024-04-26T20:35:02.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.936+0000 7f0fb2800640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fb4102980 0x7f0fb419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:02.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.936+0000 7f0fb3200640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb4104530 0x7f0fb41a2390 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009fd0 tx=0x7f0fac00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:02.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.936+0000 7f0fab400640 1 -- 172.21.15.102:0/1538908535 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac00ce20 con 0x7f0fb4104530 2024-04-26T20:35:02.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.936+0000 7f0fab400640 1 -- 172.21.15.102:0/1538908535 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fac005020 con 0x7f0fb4104530 2024-04-26T20:35:02.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.936+0000 7f0fab400640 1 -- 172.21.15.102:0/1538908535 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac010690 con 0x7f0fb4104530 2024-04-26T20:35:02.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.936+0000 7f0fb9400640 1 -- 172.21.15.102:0/1538908535 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb41a2cf0 con 0x7f0fb4104530 2024-04-26T20:35:02.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.936+0000 7f0fb9400640 1 -- 172.21.15.102:0/1538908535 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb41a31c0 con 0x7f0fb4104530 2024-04-26T20:35:02.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.938+0000 7f0fb9400640 1 -- 172.21.15.102:0/1538908535 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb41a33f0 con 0x7f0fb4104530 2024-04-26T20:35:02.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.940+0000 7f0fab400640 1 -- 172.21.15.102:0/1538908535 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f0fac0054d0 con 0x7f0fb4104530 2024-04-26T20:35:02.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.941+0000 7f0fab400640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0f980a3c40 0x7f0f980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:02.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.941+0000 7f0fab400640 1 -- 172.21.15.102:0/1538908535 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f0fac0d0ff0 con 0x7f0fb4104530 2024-04-26T20:35:02.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.941+0000 7f0fb2800640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0f980a3c40 0x7f0f980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:02.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.942+0000 7f0fb2800640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0f980a3c40 0x7f0f980a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40058c0 tx=0x7f0fa4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:02.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:02.946+0000 7f0fab400640 1 -- 172.21.15.102:0/1538908535 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0fac09d4a0 con 0x7f0fb4104530 2024-04-26T20:35:03.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.149+0000 7f0fb9400640 1 -- 172.21.15.102:0/1538908535 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0fb41a35c0 con 0x7f0fb4104530 2024-04-26T20:35:03.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.150+0000 7f0fab400640 1 -- 172.21.15.102:0/1538908535 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f0fac09cda0 con 0x7f0fb4104530 2024-04-26T20:35:03.151 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.152+0000 7f0fb9400640 1 -- 172.21.15.102:0/1538908535 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0f980a3c40 msgr2=0x7f0f980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.152+0000 7f0fb9400640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0f980a3c40 0x7f0f980a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40058c0 tx=0x7f0fa4005f90 comp rx=0 tx=0).stop 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.152+0000 7f0fb9400640 1 -- 172.21.15.102:0/1538908535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb4104530 msgr2=0x7f0fb41a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.152+0000 7f0fb9400640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb4104530 0x7f0fb41a2390 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009fd0 tx=0x7f0fac00ef10 comp rx=0 tx=0).stop 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.153+0000 7f0fb9400640 1 -- 172.21.15.102:0/1538908535 shutdown_connections 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.153+0000 7f0fb9400640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0f980a3c40 0x7f0f980a6100 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.153+0000 7f0fb9400640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb4104530 0x7f0fb41a2390 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.153+0000 7f0fb9400640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb4103b70 0x7f0fb419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.153+0000 7f0fb9400640 1 --2- 172.21.15.102:0/1538908535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0fb4102980 0x7f0fb419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.153+0000 7f0fb9400640 1 -- 172.21.15.102:0/1538908535 >> 172.21.15.102:0/1538908535 conn(0x7f0fb40fdf10 msgr2=0x7f0fb4074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:03.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.153+0000 7f0fb9400640 1 -- 172.21.15.102:0/1538908535 shutdown_connections 2024-04-26T20:35:03.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.153+0000 7f0fb9400640 1 -- 172.21.15.102:0/1538908535 wait complete. 2024-04-26T20:35:03.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:35:03.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.365+0000 7f55f4a00640 1 -- 172.21.15.102:0/3970586341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f0103ab0 msgr2=0x7f55f0103f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:03.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.365+0000 7f55f4a00640 1 --2- 172.21.15.102:0/3970586341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f0103ab0 0x7f55f0103f30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009860 tx=0x7f55dc02a340 comp rx=0 tx=0).stop 2024-04-26T20:35:03.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.365+0000 7f55f4a00640 1 -- 172.21.15.102:0/3970586341 shutdown_connections 2024-04-26T20:35:03.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.365+0000 7f55f4a00640 1 --2- 172.21.15.102:0/3970586341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55f0104470 0x7f55f010ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.365+0000 7f55f4a00640 1 --2- 172.21.15.102:0/3970586341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f0103ab0 0x7f55f0103f30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.365+0000 7f55f4a00640 1 --2- 172.21.15.102:0/3970586341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55f01028c0 0x7f55f0102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.365+0000 7f55f4a00640 1 -- 172.21.15.102:0/3970586341 >> 172.21.15.102:0/3970586341 conn(0x7f55f00fde30 msgr2=0x7f55f0100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:03.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.365+0000 7f55f4a00640 1 -- 172.21.15.102:0/3970586341 shutdown_connections 2024-04-26T20:35:03.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.365+0000 7f55f4a00640 1 -- 172.21.15.102:0/3970586341 wait complete. 2024-04-26T20:35:03.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.366+0000 7f55f4a00640 1 Processor -- start 2024-04-26T20:35:03.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55f4a00640 1 -- start start 2024-04-26T20:35:03.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55f4a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55f01028c0 0x7f55f019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:03.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55f4a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f0103ab0 0x7f55f019b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:03.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55f4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55f0104470 0x7f55f01a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:03.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55f4a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f55f010e360 con 0x7f55f0104470 2024-04-26T20:35:03.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55f4a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55f010e1e0 con 0x7f55f0103ab0 2024-04-26T20:35:03.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55f4a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f55f010e4e0 con 0x7f55f01028c0 2024-04-26T20:35:03.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55eea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f0103ab0 0x7f55f019b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:03.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55eea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f0103ab0 0x7f55f019b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:44706/0 (socket says 172.21.15.102:44706) 2024-04-26T20:35:03.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55eea00640 1 -- 172.21.15.102:0/2540139279 learned_addr learned my addr 172.21.15.102:0/2540139279 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:03.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.367+0000 7f55efe00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55f0104470 0x7f55f01a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:03.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.368+0000 7f55ef400640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55f01028c0 0x7f55f019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:03.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.368+0000 7f55eea00640 1 -- 172.21.15.102:0/2540139279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55f01028c0 msgr2=0x7f55f019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:03.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.368+0000 7f55eea00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55f01028c0 0x7f55f019aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.368+0000 7f55eea00640 1 -- 172.21.15.102:0/2540139279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55f0104470 msgr2=0x7f55f01a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:03.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.368+0000 7f55eea00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55f0104470 0x7f55f01a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.368+0000 7f55eea00640 1 -- 172.21.15.102:0/2540139279 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f01a28e0 con 0x7f55f0103ab0 2024-04-26T20:35:03.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.368+0000 7f55eea00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f0103ab0 0x7f55f019b1e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0057c0 tx=0x7f55dc032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:03.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.368+0000 7f55e7e00640 1 -- 172.21.15.102:0/2540139279 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc037040 con 0x7f55f0103ab0 2024-04-26T20:35:03.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.369+0000 7f55e7e00640 1 -- 172.21.15.102:0/2540139279 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55dc030bd0 con 0x7f55f0103ab0 2024-04-26T20:35:03.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.369+0000 7f55e7e00640 1 -- 172.21.15.102:0/2540139279 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc005280 con 0x7f55f0103ab0 2024-04-26T20:35:03.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.369+0000 7f55f4a00640 1 -- 172.21.15.102:0/2540139279 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f01a2b70 con 0x7f55f0103ab0 2024-04-26T20:35:03.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.369+0000 7f55f4a00640 1 -- 172.21.15.102:0/2540139279 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f01a2fb0 con 0x7f55f0103ab0 2024-04-26T20:35:03.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.370+0000 7f55f4a00640 1 -- 172.21.15.102:0/2540139279 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55f01a31e0 con 0x7f55f0103ab0 2024-04-26T20:35:03.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.372+0000 7f55e7e00640 1 -- 172.21.15.102:0/2540139279 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f55dc043020 con 0x7f55f0103ab0 2024-04-26T20:35:03.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.374+0000 7f55e7e00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55d40a3cc0 0x7f55d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:03.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.374+0000 7f55ef400640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55d40a3cc0 0x7f55d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:03.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.374+0000 7f55e7e00640 1 -- 172.21.15.102:0/2540139279 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f55dc0efe30 con 0x7f55f0103ab0 2024-04-26T20:35:03.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.374+0000 7f55ef400640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55d40a3cc0 0x7f55d40a6180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f55f019bb70 tx=0x7f55e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:03.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.378+0000 7f55e7e00640 1 -- 172.21.15.102:0/2540139279 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f55dc0bc3b0 con 0x7f55f0103ab0 2024-04-26T20:35:03.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.579+0000 7f55f4a00640 1 -- 172.21.15.102:0/2540139279 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f55f01a3570 con 0x7f55f0103ab0 2024-04-26T20:35:03.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.580+0000 7f55e7e00640 1 -- 172.21.15.102:0/2540139279 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f55dc0bbcb0 con 0x7f55f0103ab0 2024-04-26T20:35:03.581 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:35:03.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 -- 172.21.15.102:0/2540139279 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55d40a3cc0 msgr2=0x7f55d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:03.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55d40a3cc0 0x7f55d40a6180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f55f019bb70 tx=0x7f55e0005f90 comp rx=0 tx=0).stop 2024-04-26T20:35:03.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 -- 172.21.15.102:0/2540139279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f0103ab0 msgr2=0x7f55f019b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:03.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f0103ab0 0x7f55f019b1e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0057c0 tx=0x7f55dc032da0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 -- 172.21.15.102:0/2540139279 shutdown_connections 2024-04-26T20:35:03.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f55d40a3cc0 0x7f55d40a6180 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55f0104470 0x7f55f01a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55f0103ab0 0x7f55f019b1e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 --2- 172.21.15.102:0/2540139279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55f01028c0 0x7f55f019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 -- 172.21.15.102:0/2540139279 >> 172.21.15.102:0/2540139279 conn(0x7f55f00fde30 msgr2=0x7f55f0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:03.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 -- 172.21.15.102:0/2540139279 shutdown_connections 2024-04-26T20:35:03.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.583+0000 7f55f4a00640 1 -- 172.21.15.102:0/2540139279 wait complete. 2024-04-26T20:35:03.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:35:03.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.808+0000 7f2ad1c00640 1 -- 172.21.15.102:0/733615991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2acc106540 msgr2=0x7f2acc10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:03.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.808+0000 7f2ad1c00640 1 --2- 172.21.15.102:0/733615991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2acc106540 0x7f2acc10cdd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009ca0 tx=0x7f2ac402a8b0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.809+0000 7f2ad1c00640 1 -- 172.21.15.102:0/733615991 shutdown_connections 2024-04-26T20:35:03.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.809+0000 7f2ad1c00640 1 --2- 172.21.15.102:0/733615991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2acc106540 0x7f2acc10cdd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.809+0000 7f2ad1c00640 1 --2- 172.21.15.102:0/733615991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2acc105b80 0x7f2acc106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.809+0000 7f2ad1c00640 1 --2- 172.21.15.102:0/733615991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2acc104990 0x7f2acc104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.809+0000 7f2ad1c00640 1 -- 172.21.15.102:0/733615991 >> 172.21.15.102:0/733615991 conn(0x7f2acc0fff00 msgr2=0x7f2acc102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.809+0000 7f2ad1c00640 1 -- 172.21.15.102:0/733615991 shutdown_connections 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.809+0000 7f2ad1c00640 1 -- 172.21.15.102:0/733615991 wait complete. 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2ad1c00640 1 Processor -- start 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2ad1c00640 1 -- start start 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2ad1c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2acc104990 0x7f2acc19ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2ad1c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2acc105b80 0x7f2acc19b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2ad1c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2acc106540 0x7f2acc1a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2ad1c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc1104c0 con 0x7f2acc104990 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2ad1c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc110340 con 0x7f2acc106540 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2ad1c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2acc110640 con 0x7f2acc105b80 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2acbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2acc104990 0x7f2acc19ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2acbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2acc104990 0x7f2acc19ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52978/0 (socket says 172.21.15.102:52978) 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.810+0000 7f2acbe00640 1 -- 172.21.15.102:0/2056744607 learned_addr learned my addr 172.21.15.102:0/2056744607 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2ad1200640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2acc106540 0x7f2acc1a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2ac3400640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2acc105b80 0x7f2acc19b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2acbe00640 1 -- 172.21.15.102:0/2056744607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2acc105b80 msgr2=0x7f2acc19b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2acbe00640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2acc105b80 0x7f2acc19b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2acbe00640 1 -- 172.21.15.102:0/2056744607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2acc106540 msgr2=0x7f2acc1a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2acbe00640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2acc106540 0x7f2acc1a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2acbe00640 1 -- 172.21.15.102:0/2056744607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2acc1a2760 con 0x7f2acc104990 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2ad1200640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2acc106540 0x7f2acc1a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2acbe00640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2acc104990 0x7f2acc19ab40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00e9c0 tx=0x7f2abc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2ac9600640 1 -- 172.21.15.102:0/2056744607 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00cd80 con 0x7f2acc104990 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2ac9600640 1 -- 172.21.15.102:0/2056744607 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2abc0035f0 con 0x7f2acc104990 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.811+0000 7f2ad1c00640 1 -- 172.21.15.102:0/2056744607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2acc1a2a50 con 0x7f2acc104990 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.812+0000 7f2ac9600640 1 -- 172.21.15.102:0/2056744607 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc010690 con 0x7f2acc104990 2024-04-26T20:35:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.812+0000 7f2ad1c00640 1 -- 172.21.15.102:0/2056744607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2acc1a2f50 con 0x7f2acc104990 2024-04-26T20:35:03.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.813+0000 7f2ad1c00640 1 -- 172.21.15.102:0/2056744607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2acc10bcb0 con 0x7f2acc104990 2024-04-26T20:35:03.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.815+0000 7f2ac9600640 1 -- 172.21.15.102:0/2056744607 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2abc004000 con 0x7f2acc104990 2024-04-26T20:35:03.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.816+0000 7f2ac9600640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2ab00a3c40 0x7f2ab00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:03.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.816+0000 7f2ac9600640 1 -- 172.21.15.102:0/2056744607 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f2abc0d0ee0 con 0x7f2acc104990 2024-04-26T20:35:03.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.817+0000 7f2ac3400640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2ab00a3c40 0x7f2ab00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:03.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.817+0000 7f2ac3400640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2ab00a3c40 0x7f2ab00a6100 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2acc19c050 tx=0x7f2ab4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:03.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:03.820+0000 7f2ac9600640 1 -- 172.21.15.102:0/2056744607 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2abc09d3b0 con 0x7f2acc104990 2024-04-26T20:35:03.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:03 smithi102 ceph-mon[28094]: pgmap v400: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:35:03.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:03 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1722006810' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:35:03.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:03 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1538908535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:35:03.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:03 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2540139279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:35:04.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:03 smithi112 ceph-mon[32769]: pgmap v400: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:35:04.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:03 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1722006810' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:35:04.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:03 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1538908535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:35:04.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:03 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2540139279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:35:04.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.027+0000 7f2ad1c00640 1 -- 172.21.15.102:0/2056744607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2acc063530 con 0x7f2acc104990 2024-04-26T20:35:04.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.028+0000 7f2ac9600640 1 -- 172.21.15.102:0/2056744607 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f2abc09ccb0 con 0x7f2acc104990 2024-04-26T20:35:04.029 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-26T20:35:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 -- 172.21.15.102:0/2056744607 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2ab00a3c40 msgr2=0x7f2ab00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2ab00a3c40 0x7f2ab00a6100 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2acc19c050 tx=0x7f2ab4005f90 comp rx=0 tx=0).stop 2024-04-26T20:35:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 -- 172.21.15.102:0/2056744607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2acc104990 msgr2=0x7f2acc19ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2acc104990 0x7f2acc19ab40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00e9c0 tx=0x7f2abc00ee80 comp rx=0 tx=0).stop 2024-04-26T20:35:04.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 -- 172.21.15.102:0/2056744607 shutdown_connections 2024-04-26T20:35:04.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f2ab00a3c40 0x7f2ab00a6100 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2acc106540 0x7f2acc1a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2acc105b80 0x7f2acc19b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 --2- 172.21.15.102:0/2056744607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2acc104990 0x7f2acc19ab40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 -- 172.21.15.102:0/2056744607 >> 172.21.15.102:0/2056744607 conn(0x7f2acc0fff00 msgr2=0x7f2acc108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:04.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 -- 172.21.15.102:0/2056744607 shutdown_connections 2024-04-26T20:35:04.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.031+0000 7f2ad1c00640 1 -- 172.21.15.102:0/2056744607 wait complete. 2024-04-26T20:35:04.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:35:04.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:03 smithi130 ceph-mon[32704]: pgmap v400: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:35:04.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:03 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1722006810' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:35:04.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:03 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1538908535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:35:04.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:03 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2540139279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:35:04.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.250+0000 7fcf5be00640 1 -- 172.21.15.102:0/4268562771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf5c10a980 msgr2=0x7fcf5c1112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.250+0000 7fcf5be00640 1 --2- 172.21.15.102:0/4268562771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf5c10a980 0x7fcf5c1112a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009ca0 tx=0x7fcf5402a8c0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.250+0000 7fcf5be00640 1 -- 172.21.15.102:0/4268562771 shutdown_connections 2024-04-26T20:35:04.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.250+0000 7fcf5be00640 1 --2- 172.21.15.102:0/4268562771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf5c109fc0 0x7fcf5c10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.250+0000 7fcf5be00640 1 --2- 172.21.15.102:0/4268562771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf5c10a980 0x7fcf5c1112a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.250+0000 7fcf5be00640 1 --2- 172.21.15.102:0/4268562771 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf5c108e70 0x7fcf5c109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.250+0000 7fcf5be00640 1 -- 172.21.15.102:0/4268562771 >> 172.21.15.102:0/4268562771 conn(0x7fcf5c077e00 msgr2=0x7fcf5c07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:04.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.250+0000 7fcf5be00640 1 -- 172.21.15.102:0/4268562771 shutdown_connections 2024-04-26T20:35:04.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.250+0000 7fcf5be00640 1 -- 172.21.15.102:0/4268562771 wait complete. 2024-04-26T20:35:04.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.251+0000 7fcf5be00640 1 Processor -- start 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.251+0000 7fcf5be00640 1 -- start start 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf5c108e70 0x7fcf5c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf5c109fc0 0x7fcf5c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf5c10a980 0x7fcf5c1a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c1149c0 con 0x7fcf5c108e70 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5be00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c114840 con 0x7fcf5c10a980 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5be00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c114b40 con 0x7fcf5c109fc0 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf5c108e70 0x7fcf5c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5b400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf5c10a980 0x7fcf5c1a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf53e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf5c109fc0 0x7fcf5c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf5c108e70 0x7fcf5c19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52998/0 (socket says 172.21.15.102:52998) 2024-04-26T20:35:04.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5b400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf5c10a980 0x7fcf5c1a6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:44728/0 (socket says 172.21.15.102:44728) 2024-04-26T20:35:04.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.252+0000 7fcf5aa00640 1 -- 172.21.15.102:0/3440143352 learned_addr learned my addr 172.21.15.102:0/3440143352 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:04.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.253+0000 7fcf5b400640 1 -- 172.21.15.102:0/3440143352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf5c109fc0 msgr2=0x7fcf5c19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.253+0000 7fcf5b400640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf5c109fc0 0x7fcf5c19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.253+0000 7fcf5b400640 1 -- 172.21.15.102:0/3440143352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf5c108e70 msgr2=0x7fcf5c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.253+0000 7fcf5b400640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf5c108e70 0x7fcf5c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.253+0000 7fcf5b400640 1 -- 172.21.15.102:0/3440143352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c1a6c00 con 0x7fcf5c10a980 2024-04-26T20:35:04.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.253+0000 7fcf5b400640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf5c10a980 0x7fcf5c1a6590 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcf54031a20 tx=0x7fcf540317a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:04.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.253+0000 7fcf60a00640 1 -- 172.21.15.102:0/3440143352 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54003c30 con 0x7fcf5c10a980 2024-04-26T20:35:04.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.253+0000 7fcf5be00640 1 -- 172.21.15.102:0/3440143352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5c1a6e90 con 0x7fcf5c10a980 2024-04-26T20:35:04.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.253+0000 7fcf60a00640 1 -- 172.21.15.102:0/3440143352 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf54003dd0 con 0x7fcf5c10a980 2024-04-26T20:35:04.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.253+0000 7fcf60a00640 1 -- 172.21.15.102:0/3440143352 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5402fb20 con 0x7fcf5c10a980 2024-04-26T20:35:04.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.254+0000 7fcf5be00640 1 -- 172.21.15.102:0/3440143352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5c1a73a0 con 0x7fcf5c10a980 2024-04-26T20:35:04.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.255+0000 7fcf5be00640 1 -- 172.21.15.102:0/3440143352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf5c110180 con 0x7fcf5c10a980 2024-04-26T20:35:04.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.257+0000 7fcf60a00640 1 -- 172.21.15.102:0/3440143352 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fcf54004660 con 0x7fcf5c10a980 2024-04-26T20:35:04.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.258+0000 7fcf60a00640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fcf3c0a3c40 0x7fcf3c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:04.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.258+0000 7fcf60a00640 1 -- 172.21.15.102:0/3440143352 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fcf540f2110 con 0x7fcf5c10a980 2024-04-26T20:35:04.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.258+0000 7fcf5aa00640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fcf3c0a3c40 0x7fcf3c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:04.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.259+0000 7fcf5aa00640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fcf3c0a3c40 0x7fcf3c0a6100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009cd0 tx=0x7fcf4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:04.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.260+0000 7fcf60a00640 1 -- 172.21.15.102:0/3440143352 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcf540be720 con 0x7fcf5c10a980 2024-04-26T20:35:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.462+0000 7fcf5be00640 1 -- 172.21.15.102:0/3440143352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcf5c02dcd0 con 0x7fcf5c10a980 2024-04-26T20:35:04.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.463+0000 7fcf60a00640 1 -- 172.21.15.102:0/3440143352 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fcf540be020 con 0x7fcf5c10a980 2024-04-26T20:35:04.464 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.465+0000 7fcf5be00640 1 -- 172.21.15.102:0/3440143352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fcf3c0a3c40 msgr2=0x7fcf3c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.465+0000 7fcf5be00640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fcf3c0a3c40 0x7fcf3c0a6100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009cd0 tx=0x7fcf4c005f90 comp rx=0 tx=0).stop 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.465+0000 7fcf5be00640 1 -- 172.21.15.102:0/3440143352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf5c10a980 msgr2=0x7fcf5c1a6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.465+0000 7fcf5be00640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf5c10a980 0x7fcf5c1a6590 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcf54031a20 tx=0x7fcf540317a0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.465+0000 7fcf5be00640 1 -- 172.21.15.102:0/3440143352 shutdown_connections 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.466+0000 7fcf5be00640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fcf3c0a3c40 0x7fcf3c0a6100 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.466+0000 7fcf5be00640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf5c10a980 0x7fcf5c1a6590 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.466+0000 7fcf5be00640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf5c109fc0 0x7fcf5c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.466+0000 7fcf5be00640 1 --2- 172.21.15.102:0/3440143352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf5c108e70 0x7fcf5c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.466+0000 7fcf5be00640 1 -- 172.21.15.102:0/3440143352 >> 172.21.15.102:0/3440143352 conn(0x7fcf5c077e00 msgr2=0x7fcf5c10d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.466+0000 7fcf5be00640 1 -- 172.21.15.102:0/3440143352 shutdown_connections 2024-04-26T20:35:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.466+0000 7fcf5be00640 1 -- 172.21.15.102:0/3440143352 wait complete. 2024-04-26T20:35:04.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:35:04.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.690+0000 7fe3f4a00640 1 -- 172.21.15.102:0/2696298317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f0105d10 msgr2=0x7fe3f0108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.690+0000 7fe3f4a00640 1 --2- 172.21.15.102:0/2696298317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f0105d10 0x7fe3f0108100 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009900 tx=0x7fe3d802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.690+0000 7fe3f4a00640 1 -- 172.21.15.102:0/2696298317 shutdown_connections 2024-04-26T20:35:04.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.690+0000 7fe3f4a00640 1 --2- 172.21.15.102:0/2696298317 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3f0108640 0x7fe3f010aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.690+0000 7fe3f4a00640 1 --2- 172.21.15.102:0/2696298317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f0105d10 0x7fe3f0108100 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.690+0000 7fe3f4a00640 1 --2- 172.21.15.102:0/2696298317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3f01033e0 0x7fe3f01057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.690+0000 7fe3f4a00640 1 -- 172.21.15.102:0/2696298317 >> 172.21.15.102:0/2696298317 conn(0x7fe3f00fcdf0 msgr2=0x7fe3f00ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:04.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.690+0000 7fe3f4a00640 1 -- 172.21.15.102:0/2696298317 shutdown_connections 2024-04-26T20:35:04.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.690+0000 7fe3f4a00640 1 -- 172.21.15.102:0/2696298317 wait complete. 2024-04-26T20:35:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.691+0000 7fe3f4a00640 1 Processor -- start 2024-04-26T20:35:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3f4a00640 1 -- start start 2024-04-26T20:35:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3f4a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3f01033e0 0x7fe3f019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3f4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3f0105d10 0x7fe3f019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3f4a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f0108640 0x7fe3f01a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3f4a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f010e4e0 con 0x7fe3f0105d10 2024-04-26T20:35:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3f4a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f010e360 con 0x7fe3f01033e0 2024-04-26T20:35:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3f4a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f010e660 con 0x7fe3f0108640 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3ede00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3f01033e0 0x7fe3f019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3ede00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3f01033e0 0x7fe3f019add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:44752/0 (socket says 172.21.15.102:44752) 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3ede00640 1 -- 172.21.15.102:0/275801289 learned_addr learned my addr 172.21.15.102:0/275801289 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3e5400640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3f0105d10 0x7fe3f019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.692+0000 7fe3ee800640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f0108640 0x7fe3f01a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.693+0000 7fe3ede00640 1 -- 172.21.15.102:0/275801289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f0108640 msgr2=0x7fe3f01a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.693+0000 7fe3ede00640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f0108640 0x7fe3f01a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.693+0000 7fe3ede00640 1 -- 172.21.15.102:0/275801289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3f0105d10 msgr2=0x7fe3f019b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.693+0000 7fe3ede00640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3f0105d10 0x7fe3f019b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.693+0000 7fe3ede00640 1 -- 172.21.15.102:0/275801289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f01a29f0 con 0x7fe3f01033e0 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.693+0000 7fe3ee800640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f0108640 0x7fe3f01a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.693+0000 7fe3ede00640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3f01033e0 0x7fe3f019add0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000d950 tx=0x7fe3e000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.693+0000 7fe3e7400640 1 -- 172.21.15.102:0/275801289 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0017070 con 0x7fe3f01033e0 2024-04-26T20:35:04.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.693+0000 7fe3e7400640 1 -- 172.21.15.102:0/275801289 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e0009c30 con 0x7fe3f01033e0 2024-04-26T20:35:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.693+0000 7fe3f4a00640 1 -- 172.21.15.102:0/275801289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f01a2ce0 con 0x7fe3f01033e0 2024-04-26T20:35:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.694+0000 7fe3e7400640 1 -- 172.21.15.102:0/275801289 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e000f620 con 0x7fe3f01033e0 2024-04-26T20:35:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.694+0000 7fe3f4a00640 1 -- 172.21.15.102:0/275801289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f01a31b0 con 0x7fe3f01033e0 2024-04-26T20:35:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.695+0000 7fe3f4a00640 1 -- 172.21.15.102:0/275801289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3f010a190 con 0x7fe3f01033e0 2024-04-26T20:35:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.697+0000 7fe3e7400640 1 -- 172.21.15.102:0/275801289 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fe3e0010460 con 0x7fe3f01033e0 2024-04-26T20:35:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.698+0000 7fe3e7400640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe3d40a3c40 0x7fe3d40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.698+0000 7fe3e7400640 1 -- 172.21.15.102:0/275801289 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fe3e00d0520 con 0x7fe3f01033e0 2024-04-26T20:35:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.699+0000 7fe3e5400640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe3d40a3c40 0x7fe3d40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.699+0000 7fe3e5400640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe3d40a3c40 0x7fe3d40a6100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8005ab0 tx=0x7fe3d8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.704+0000 7fe3e7400640 1 -- 172.21.15.102:0/275801289 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe3e0009de0 con 0x7fe3f01033e0 2024-04-26T20:35:04.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:04 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2056744607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:35:04.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:04 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3440143352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:35:04.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.950+0000 7fe3f4a00640 1 -- 172.21.15.102:0/275801289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3f002dd40 con 0x7fe3f01033e0 2024-04-26T20:35:04.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.952+0000 7fe3e7400640 1 -- 172.21.15.102:0/275801289 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11954 (secure 0 0 0) 0x7fe3e009c340 con 0x7fe3f01033e0 2024-04-26T20:35:04.953 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:35:04.953 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:32:25.769708+0000","modified":"2024-04-26T20:32:55.699799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24593,"mds_1":24565,"mds_2":24547,"mds_3":24533,"mds_4":24526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24565,24593,24547,24533,24526]},"id":1}]} 2024-04-26T20:35:04.954 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-26T20:35:04.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 -- 172.21.15.102:0/275801289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe3d40a3c40 msgr2=0x7fe3d40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe3d40a3c40 0x7fe3d40a6100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8005ab0 tx=0x7fe3d8005a20 comp rx=0 tx=0).stop 2024-04-26T20:35:04.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 -- 172.21.15.102:0/275801289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3f01033e0 msgr2=0x7fe3f019add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:04.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3f01033e0 0x7fe3f019add0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000d950 tx=0x7fe3e000de10 comp rx=0 tx=0).stop 2024-04-26T20:35:04.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 -- 172.21.15.102:0/275801289 shutdown_connections 2024-04-26T20:35:04.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe3d40a3c40 0x7fe3d40a6100 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3f0105d10 0x7fe3f019b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f0108640 0x7fe3f01a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 --2- 172.21.15.102:0/275801289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3f01033e0 0x7fe3f019add0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:04.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 -- 172.21.15.102:0/275801289 >> 172.21.15.102:0/275801289 conn(0x7fe3f00fcdf0 msgr2=0x7fe3f0102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:04.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 -- 172.21.15.102:0/275801289 shutdown_connections 2024-04-26T20:35:04.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:04.955+0000 7fe3f4a00640 1 -- 172.21.15.102:0/275801289 wait complete. 2024-04-26T20:35:04.975 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:35:05.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:04 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2056744607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:35:05.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:04 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3440143352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:35:05.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:04 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2056744607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:35:05.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:04 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3440143352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:35:05.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.173+0000 7f8823e00640 1 -- 172.21.15.102:0/269392668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8824104980 msgr2=0x7f8824104da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:05.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.173+0000 7f8823e00640 1 --2- 172.21.15.102:0/269392668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8824104980 0x7f8824104da0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8814009900 tx=0x7f881402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.173+0000 7f8823e00640 1 -- 172.21.15.102:0/269392668 shutdown_connections 2024-04-26T20:35:05.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.173+0000 7f8823e00640 1 --2- 172.21.15.102:0/269392668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8824105b70 0x7f8824105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.174+0000 7f8823e00640 1 --2- 172.21.15.102:0/269392668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8824106530 0x7f882410cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.174+0000 7f8823e00640 1 --2- 172.21.15.102:0/269392668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8824104980 0x7f8824104da0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.174+0000 7f8823e00640 1 -- 172.21.15.102:0/269392668 >> 172.21.15.102:0/269392668 conn(0x7f88240fff10 msgr2=0x7f8824102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:05.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.174+0000 7f8823e00640 1 -- 172.21.15.102:0/269392668 shutdown_connections 2024-04-26T20:35:05.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.174+0000 7f8823e00640 1 -- 172.21.15.102:0/269392668 wait complete. 2024-04-26T20:35:05.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.175+0000 7f8823e00640 1 Processor -- start 2024-04-26T20:35:05.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.175+0000 7f8823e00640 1 -- start start 2024-04-26T20:35:05.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.175+0000 7f8823e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8824104980 0x7f882419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:05.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.175+0000 7f8823e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8824105b70 0x7f882419b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:05.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.175+0000 7f8823e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8824106530 0x7f88241a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:05.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.175+0000 7f8823e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f88241104b0 con 0x7f8824104980 2024-04-26T20:35:05.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.175+0000 7f8823e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8824110330 con 0x7f8824105b70 2024-04-26T20:35:05.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.175+0000 7f8823e00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8824110630 con 0x7f8824106530 2024-04-26T20:35:05.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8822a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8824104980 0x7f882419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:05.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8823400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8824106530 0x7f88241a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:05.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8822a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8824104980 0x7f882419abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43514/0 (socket says 172.21.15.102:43514) 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8822a00640 1 -- 172.21.15.102:0/4081661444 learned_addr learned my addr 172.21.15.102:0/4081661444 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8822000640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8824105b70 0x7f882419b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8822a00640 1 -- 172.21.15.102:0/4081661444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8824106530 msgr2=0x7f88241a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8822a00640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8824106530 0x7f88241a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8822a00640 1 -- 172.21.15.102:0/4081661444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8824105b70 msgr2=0x7f882419b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8822a00640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8824105b70 0x7f882419b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8822a00640 1 -- 172.21.15.102:0/4081661444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88241a27d0 con 0x7f8824104980 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8823400640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8824106530 0x7f88241a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.176+0000 7f8822a00640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8824104980 0x7f882419abb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8814006010 tx=0x7f88140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.177+0000 7f881b400640 1 -- 172.21.15.102:0/4081661444 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881403f030 con 0x7f8824104980 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.177+0000 7f881b400640 1 -- 172.21.15.102:0/4081661444 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88140318f0 con 0x7f8824104980 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.177+0000 7f8823e00640 1 -- 172.21.15.102:0/4081661444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88241a2a60 con 0x7f8824104980 2024-04-26T20:35:05.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.177+0000 7f881b400640 1 -- 172.21.15.102:0/4081661444 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8814035070 con 0x7f8824104980 2024-04-26T20:35:05.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.177+0000 7f8823e00640 1 -- 172.21.15.102:0/4081661444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88241a2f50 con 0x7f8824104980 2024-04-26T20:35:05.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.178+0000 7f8823e00640 1 -- 172.21.15.102:0/4081661444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f882410bd70 con 0x7f8824104980 2024-04-26T20:35:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.180+0000 7f881b400640 1 -- 172.21.15.102:0/4081661444 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8814042030 con 0x7f8824104980 2024-04-26T20:35:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.182+0000 7f881b400640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88080a3c40 0x7f88080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.182+0000 7f881b400640 1 -- 172.21.15.102:0/4081661444 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f88140f1230 con 0x7f8824104980 2024-04-26T20:35:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.182+0000 7f8822000640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88080a3c40 0x7f88080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:05.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.182+0000 7f8822000640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88080a3c40 0x7f88080a6100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f882419c0c0 tx=0x7f881000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:05.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.184+0000 7f881b400640 1 -- 172.21.15.102:0/4081661444 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f88140bd050 con 0x7f8824104980 2024-04-26T20:35:05.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.388+0000 7f8823e00640 1 -- 172.21.15.102:0/4081661444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f882402dca0 con 0x7f8824104980 2024-04-26T20:35:05.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.391+0000 7f881b400640 1 -- 172.21.15.102:0/4081661444 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24833 (secure 0 0 0) 0x7f88140c1350 con 0x7f8824104980 2024-04-26T20:35:05.392 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:35:05.392 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:34:30.697401+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:35:05.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.393+0000 7f8823e00640 1 -- 172.21.15.102:0/4081661444 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88080a3c40 msgr2=0x7f88080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.393+0000 7f8823e00640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88080a3c40 0x7f88080a6100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f882419c0c0 tx=0x7f881000b390 comp rx=0 tx=0).stop 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.393+0000 7f8823e00640 1 -- 172.21.15.102:0/4081661444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8824104980 msgr2=0x7f882419abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.393+0000 7f8823e00640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8824104980 0x7f882419abb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8814006010 tx=0x7f88140042f0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.393+0000 7f8823e00640 1 -- 172.21.15.102:0/4081661444 shutdown_connections 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.393+0000 7f8823e00640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f88080a3c40 0x7f88080a6100 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.393+0000 7f8823e00640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8824106530 0x7f88241a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.393+0000 7f8823e00640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8824105b70 0x7f882419b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.393+0000 7f8823e00640 1 --2- 172.21.15.102:0/4081661444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8824104980 0x7f882419abb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.393+0000 7f8823e00640 1 -- 172.21.15.102:0/4081661444 >> 172.21.15.102:0/4081661444 conn(0x7f88240fff10 msgr2=0x7f8824108f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.394+0000 7f8823e00640 1 -- 172.21.15.102:0/4081661444 shutdown_connections 2024-04-26T20:35:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.394+0000 7f8823e00640 1 -- 172.21.15.102:0/4081661444 wait complete. 2024-04-26T20:35:05.413 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:27:10.527933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:32:23.439237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:32:24.702263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:32:28.348785+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:35:05.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:35:05.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.619+0000 7f7de8a00640 1 -- 172.21.15.102:0/507606011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7de410a980 msgr2=0x7f7de41112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:05.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.619+0000 7f7de8a00640 1 --2- 172.21.15.102:0/507606011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7de410a980 0x7f7de41112a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009ca0 tx=0x7f7ddc02a8c0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.619+0000 7f7de8a00640 1 -- 172.21.15.102:0/507606011 shutdown_connections 2024-04-26T20:35:05.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.619+0000 7f7de8a00640 1 --2- 172.21.15.102:0/507606011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7de4109fc0 0x7f7de410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.619+0000 7f7de8a00640 1 --2- 172.21.15.102:0/507606011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7de410a980 0x7f7de41112a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.619+0000 7f7de8a00640 1 --2- 172.21.15.102:0/507606011 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7de4108e70 0x7f7de4109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.619+0000 7f7de8a00640 1 -- 172.21.15.102:0/507606011 >> 172.21.15.102:0/507606011 conn(0x7f7de4077e00 msgr2=0x7f7de407a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:05.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.619+0000 7f7de8a00640 1 -- 172.21.15.102:0/507606011 shutdown_connections 2024-04-26T20:35:05.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.619+0000 7f7de8a00640 1 -- 172.21.15.102:0/507606011 wait complete. 2024-04-26T20:35:05.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.620+0000 7f7de8a00640 1 Processor -- start 2024-04-26T20:35:05.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.620+0000 7f7de8a00640 1 -- start start 2024-04-26T20:35:05.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de8a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7de4108e70 0x7f7de419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:05.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de8a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7de4109fc0 0x7f7de419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de8a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7de410a980 0x7f7de41a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de8a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4114a10 con 0x7f7de4108e70 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de8a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4114890 con 0x7f7de410a980 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de8a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4114b90 con 0x7f7de4109fc0 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7de4108e70 0x7f7de419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7de4108e70 0x7f7de419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43528/0 (socket says 172.21.15.102:43528) 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7dd9e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7de4109fc0 0x7f7de419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de2800640 1 -- 172.21.15.102:0/2053335074 learned_addr learned my addr 172.21.15.102:0/2053335074 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de3200640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7de410a980 0x7f7de41a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de2800640 1 -- 172.21.15.102:0/2053335074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7de4109fc0 msgr2=0x7f7de419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de2800640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7de4109fc0 0x7f7de419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de2800640 1 -- 172.21.15.102:0/2053335074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7de410a980 msgr2=0x7f7de41a6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:05.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de2800640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7de410a980 0x7f7de41a6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.621+0000 7f7de2800640 1 -- 172.21.15.102:0/2053335074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de41a6c90 con 0x7f7de4108e70 2024-04-26T20:35:05.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.622+0000 7f7dd9e00640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7de4109fc0 0x7f7de419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:05.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.622+0000 7f7de2800640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7de4108e70 0x7f7de419f070 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400e9f0 tx=0x7f7dd400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:05.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.622+0000 7f7ddbe00640 1 -- 172.21.15.102:0/2053335074 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd400cd80 con 0x7f7de4108e70 2024-04-26T20:35:05.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.622+0000 7f7de8a00640 1 -- 172.21.15.102:0/2053335074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de41a6f80 con 0x7f7de4108e70 2024-04-26T20:35:05.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.622+0000 7f7ddbe00640 1 -- 172.21.15.102:0/2053335074 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dd40054d0 con 0x7f7de4108e70 2024-04-26T20:35:05.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.622+0000 7f7ddbe00640 1 -- 172.21.15.102:0/2053335074 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4010690 con 0x7f7de4108e70 2024-04-26T20:35:05.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.622+0000 7f7de8a00640 1 -- 172.21.15.102:0/2053335074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de41a73e0 con 0x7f7de4108e70 2024-04-26T20:35:05.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.623+0000 7f7de8a00640 1 -- 172.21.15.102:0/2053335074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7de4110120 con 0x7f7de4108e70 2024-04-26T20:35:05.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.625+0000 7f7ddbe00640 1 -- 172.21.15.102:0/2053335074 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f7dd4004e70 con 0x7f7de4108e70 2024-04-26T20:35:05.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.627+0000 7f7ddbe00640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7dc80a3c40 0x7f7dc80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:05.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.627+0000 7f7ddbe00640 1 -- 172.21.15.102:0/2053335074 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f7dd40d0d80 con 0x7f7de4108e70 2024-04-26T20:35:05.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.627+0000 7f7dd9e00640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7dc80a3c40 0x7f7dc80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:05.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.628+0000 7f7dd9e00640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7dc80a3c40 0x7f7dc80a6100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7de41a0580 tx=0x7f7dcc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:05.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.631+0000 7f7ddbe00640 1 -- 172.21.15.102:0/2053335074 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7dd409d1f0 con 0x7f7de4108e70 2024-04-26T20:35:05.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.820+0000 7f7de8a00640 1 -- 172.21.15.102:0/2053335074 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7de402dd20 con 0x7f7de4108e70 2024-04-26T20:35:05.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.821+0000 7f7ddbe00640 1 -- 172.21.15.102:0/2053335074 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f7dd409caf0 con 0x7f7de4108e70 2024-04-26T20:35:05.822 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.823+0000 7f7de8a00640 1 -- 172.21.15.102:0/2053335074 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7dc80a3c40 msgr2=0x7f7dc80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.823+0000 7f7de8a00640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7dc80a3c40 0x7f7dc80a6100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7de41a0580 tx=0x7f7dcc005f90 comp rx=0 tx=0).stop 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.823+0000 7f7de8a00640 1 -- 172.21.15.102:0/2053335074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7de4108e70 msgr2=0x7f7de419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.823+0000 7f7de8a00640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7de4108e70 0x7f7de419f070 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400e9f0 tx=0x7f7dd400edf0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.823+0000 7f7de8a00640 1 -- 172.21.15.102:0/2053335074 shutdown_connections 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.824+0000 7f7de8a00640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f7dc80a3c40 0x7f7dc80a6100 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.824+0000 7f7de8a00640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7de410a980 0x7f7de41a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.824+0000 7f7de8a00640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7de4109fc0 0x7f7de419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.824+0000 7f7de8a00640 1 --2- 172.21.15.102:0/2053335074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7de4108e70 0x7f7de419f070 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.824+0000 7f7de8a00640 1 -- 172.21.15.102:0/2053335074 >> 172.21.15.102:0/2053335074 conn(0x7f7de4077e00 msgr2=0x7f7de410d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.824+0000 7f7de8a00640 1 -- 172.21.15.102:0/2053335074 shutdown_connections 2024-04-26T20:35:05.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:05.824+0000 7f7de8a00640 1 -- 172.21.15.102:0/2053335074 wait complete. 2024-04-26T20:35:05.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:35:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:05 smithi112 ceph-mon[32769]: pgmap v401: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:35:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:05 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/275801289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:35:06.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:05 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/4081661444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:35:06.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.034+0000 7f2416400640 1 -- 172.21.15.102:0/3853108941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2410104b90 msgr2=0x7f2410104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.034+0000 7f2416400640 1 --2- 172.21.15.102:0/3853108941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2410104b90 0x7f2410104fb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2400009900 tx=0x7f240002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.034+0000 7f2416400640 1 -- 172.21.15.102:0/3853108941 shutdown_connections 2024-04-26T20:35:06.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.034+0000 7f2416400640 1 --2- 172.21.15.102:0/3853108941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2410106740 0x7f241010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.034+0000 7f2416400640 1 --2- 172.21.15.102:0/3853108941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2410105d80 0x7f2410106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.034+0000 7f2416400640 1 --2- 172.21.15.102:0/3853108941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2410104b90 0x7f2410104fb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.034+0000 7f2416400640 1 -- 172.21.15.102:0/3853108941 >> 172.21.15.102:0/3853108941 conn(0x7f24101000e0 msgr2=0x7f2410102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:06.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.034+0000 7f2416400640 1 -- 172.21.15.102:0/3853108941 shutdown_connections 2024-04-26T20:35:06.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.034+0000 7f2416400640 1 -- 172.21.15.102:0/3853108941 wait complete. 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.035+0000 7f2416400640 1 Processor -- start 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.035+0000 7f2416400640 1 -- start start 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.035+0000 7f2416400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2410104b90 0x7f241019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.035+0000 7f2416400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2410105d80 0x7f241019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.035+0000 7f2416400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2410106740 0x7f24101a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.035+0000 7f2416400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24101106e0 con 0x7f2410104b90 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.035+0000 7f2416400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2410110560 con 0x7f2410105d80 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.035+0000 7f2416400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2410110860 con 0x7f2410106740 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f240f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2410104b90 0x7f241019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f240f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2410104b90 0x7f241019adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43546/0 (socket says 172.21.15.102:43546) 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f240f400640 1 -- 172.21.15.102:0/2755729068 learned_addr learned my addr 172.21.15.102:0/2755729068 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:06.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f240fe00640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2410106740 0x7f24101a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f2407e00640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2410105d80 0x7f241019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f240f400640 1 -- 172.21.15.102:0/2755729068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2410106740 msgr2=0x7f24101a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f240f400640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2410106740 0x7f24101a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f240f400640 1 -- 172.21.15.102:0/2755729068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2410105d80 msgr2=0x7f241019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f240f400640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2410105d80 0x7f241019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f240f400640 1 -- 172.21.15.102:0/2755729068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24101a29e0 con 0x7f2410104b90 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f2407e00640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2410105d80 0x7f241019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.036+0000 7f240f400640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2410104b90 0x7f241019adc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2400031500 tx=0x7f2400003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.037+0000 7f240cc00640 1 -- 172.21.15.102:0/2755729068 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240003f070 con 0x7f2410104b90 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.037+0000 7f240cc00640 1 -- 172.21.15.102:0/2755729068 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2400031c50 con 0x7f2410104b90 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.037+0000 7f2416400640 1 -- 172.21.15.102:0/2755729068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24101a2c70 con 0x7f2410104b90 2024-04-26T20:35:06.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.037+0000 7f240cc00640 1 -- 172.21.15.102:0/2755729068 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2400035030 con 0x7f2410104b90 2024-04-26T20:35:06.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.037+0000 7f2416400640 1 -- 172.21.15.102:0/2755729068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24101a3160 con 0x7f2410104b90 2024-04-26T20:35:06.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.038+0000 7f2416400640 1 -- 172.21.15.102:0/2755729068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f241010bf30 con 0x7f2410104b90 2024-04-26T20:35:06.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.040+0000 7f240cc00640 1 -- 172.21.15.102:0/2755729068 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2400042070 con 0x7f2410104b90 2024-04-26T20:35:06.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.042+0000 7f240cc00640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f23f40a3c40 0x7f23f40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.042+0000 7f240cc00640 1 -- 172.21.15.102:0/2755729068 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f24000f10a0 con 0x7f2410104b90 2024-04-26T20:35:06.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.042+0000 7f2407e00640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f23f40a3c40 0x7f23f40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.043+0000 7f2407e00640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f23f40a3c40 0x7f23f40a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f23f80095f0 tx=0x7f23f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:06.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.046+0000 7f240cc00640 1 -- 172.21.15.102:0/2755729068 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f24000bcec0 con 0x7f2410104b90 2024-04-26T20:35:06.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:05 smithi130 ceph-mon[32704]: pgmap v401: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:35:06.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:05 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/275801289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:35:06.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:05 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/4081661444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:35:06.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:05 smithi102 ceph-mon[28094]: pgmap v401: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:35:06.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:05 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/275801289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:35:06.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:05 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/4081661444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:35:06.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.254+0000 7f2416400640 1 -- 172.21.15.102:0/2755729068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f241002dca0 con 0x7f2410104b90 2024-04-26T20:35:06.256 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:35:06.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.255+0000 7f240cc00640 1 -- 172.21.15.102:0/2755729068 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f24000c11c0 con 0x7f2410104b90 2024-04-26T20:35:06.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 -- 172.21.15.102:0/2755729068 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f23f40a3c40 msgr2=0x7f23f40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f23f40a3c40 0x7f23f40a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f23f80095f0 tx=0x7f23f8005f90 comp rx=0 tx=0).stop 2024-04-26T20:35:06.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 -- 172.21.15.102:0/2755729068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2410104b90 msgr2=0x7f241019adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2410104b90 0x7f241019adc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2400031500 tx=0x7f2400003c10 comp rx=0 tx=0).stop 2024-04-26T20:35:06.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 -- 172.21.15.102:0/2755729068 shutdown_connections 2024-04-26T20:35:06.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f23f40a3c40 0x7f23f40a6100 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2410106740 0x7f24101a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2410105d80 0x7f241019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 --2- 172.21.15.102:0/2755729068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2410104b90 0x7f241019adc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 -- 172.21.15.102:0/2755729068 >> 172.21.15.102:0/2755729068 conn(0x7f24101000e0 msgr2=0x7f2410109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:06.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 -- 172.21.15.102:0/2755729068 shutdown_connections 2024-04-26T20:35:06.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.258+0000 7f2416400640 1 -- 172.21.15.102:0/2755729068 wait complete. 2024-04-26T20:35:06.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:35:06.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.475+0000 7f5c33200640 1 -- 172.21.15.102:0/1062073430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c2c101650 msgr2=0x7f5c2c101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.475+0000 7f5c33200640 1 --2- 172.21.15.102:0/1062073430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c2c101650 0x7f5c2c101a70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009900 tx=0x7f5c2002a3f0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.475+0000 7f5c33200640 1 -- 172.21.15.102:0/1062073430 shutdown_connections 2024-04-26T20:35:06.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.475+0000 7f5c33200640 1 --2- 172.21.15.102:0/1062073430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c2c103200 0x7f5c2c109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.475+0000 7f5c33200640 1 --2- 172.21.15.102:0/1062073430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c2c102840 0x7f5c2c102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.475+0000 7f5c33200640 1 --2- 172.21.15.102:0/1062073430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c2c101650 0x7f5c2c101a70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.475+0000 7f5c33200640 1 -- 172.21.15.102:0/1062073430 >> 172.21.15.102:0/1062073430 conn(0x7f5c2c0fcc00 msgr2=0x7f5c2c0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:06.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.475+0000 7f5c33200640 1 -- 172.21.15.102:0/1062073430 shutdown_connections 2024-04-26T20:35:06.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.475+0000 7f5c33200640 1 -- 172.21.15.102:0/1062073430 wait complete. 2024-04-26T20:35:06.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.476+0000 7f5c33200640 1 Processor -- start 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.476+0000 7f5c33200640 1 -- start start 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.476+0000 7f5c33200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c2c101650 0x7f5c2c19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.476+0000 7f5c33200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c2c102840 0x7f5c2c19b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.476+0000 7f5c33200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c2c103200 0x7f5c2c1a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.476+0000 7f5c33200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c06a0d0 con 0x7f5c2c102840 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.476+0000 7f5c33200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c069f50 con 0x7f5c2c103200 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.476+0000 7f5c33200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c06a250 con 0x7f5c2c101650 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c31400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c2c102840 0x7f5c2c19b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c31e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c2c101650 0x7f5c2c19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c31400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c2c102840 0x7f5c2c19b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43566/0 (socket says 172.21.15.102:43566) 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c31400640 1 -- 172.21.15.102:0/3632381742 learned_addr learned my addr 172.21.15.102:0/3632381742 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c32800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c2c103200 0x7f5c2c1a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c31e00640 1 -- 172.21.15.102:0/3632381742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c2c103200 msgr2=0x7f5c2c1a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c31e00640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c2c103200 0x7f5c2c1a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c31e00640 1 -- 172.21.15.102:0/3632381742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c2c102840 msgr2=0x7f5c2c19b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c31e00640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c2c102840 0x7f5c2c19b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c31e00640 1 -- 172.21.15.102:0/3632381742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2c1a2800 con 0x7f5c2c101650 2024-04-26T20:35:06.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.477+0000 7f5c32800640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c2c103200 0x7f5c2c1a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:06.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.478+0000 7f5c31e00640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c2c101650 0x7f5c2c19abe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5c2002a910 tx=0x7f5c20003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:06.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.478+0000 7f5c26a00640 1 -- 172.21.15.102:0/3632381742 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2003f070 con 0x7f5c2c101650 2024-04-26T20:35:06.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.478+0000 7f5c33200640 1 -- 172.21.15.102:0/3632381742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c2c1a2a90 con 0x7f5c2c101650 2024-04-26T20:35:06.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.478+0000 7f5c26a00640 1 -- 172.21.15.102:0/3632381742 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c20031c50 con 0x7f5c2c101650 2024-04-26T20:35:06.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.478+0000 7f5c26a00640 1 -- 172.21.15.102:0/3632381742 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c20035030 con 0x7f5c2c101650 2024-04-26T20:35:06.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.478+0000 7f5c33200640 1 -- 172.21.15.102:0/3632381742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c2c1a2fa0 con 0x7f5c2c101650 2024-04-26T20:35:06.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.480+0000 7f5c33200640 1 -- 172.21.15.102:0/3632381742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c2c1a31d0 con 0x7f5c2c101650 2024-04-26T20:35:06.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.483+0000 7f5c26a00640 1 -- 172.21.15.102:0/3632381742 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f5c20042070 con 0x7f5c2c101650 2024-04-26T20:35:06.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.484+0000 7f5c26a00640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5c140a3c40 0x7f5c140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.485+0000 7f5c26a00640 1 -- 172.21.15.102:0/3632381742 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f5c200f0ff0 con 0x7f5c2c101650 2024-04-26T20:35:06.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.485+0000 7f5c31400640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5c140a3c40 0x7f5c140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.486+0000 7f5c31400640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5c140a3c40 0x7f5c140a6100 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c19c0f0 tx=0x7f5c1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:06.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.489+0000 7f5c26a00640 1 -- 172.21.15.102:0/3632381742 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5c200bce50 con 0x7f5c2c101650 2024-04-26T20:35:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.691+0000 7f5c33200640 1 -- 172.21.15.102:0/3632381742 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5c2c1a33a0 con 0x7f5c2c101650 2024-04-26T20:35:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.692+0000 7f5c26a00640 1 -- 172.21.15.102:0/3632381742 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f5c200c1150 con 0x7f5c2c101650 2024-04-26T20:35:06.694 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-26T20:35:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 -- 172.21.15.102:0/3632381742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5c140a3c40 msgr2=0x7f5c140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5c140a3c40 0x7f5c140a6100 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c19c0f0 tx=0x7f5c1c005f90 comp rx=0 tx=0).stop 2024-04-26T20:35:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 -- 172.21.15.102:0/3632381742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c2c101650 msgr2=0x7f5c2c19abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c2c101650 0x7f5c2c19abe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5c2002a910 tx=0x7f5c20003c10 comp rx=0 tx=0).stop 2024-04-26T20:35:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 -- 172.21.15.102:0/3632381742 shutdown_connections 2024-04-26T20:35:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f5c140a3c40 0x7f5c140a6100 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c2c102840 0x7f5c2c19b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c2c103200 0x7f5c2c1a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 --2- 172.21.15.102:0/3632381742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c2c101650 0x7f5c2c19abe0 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5c2002a910 tx=0x7f5c20003c10 comp rx=0 tx=0).stop 2024-04-26T20:35:06.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 -- 172.21.15.102:0/3632381742 >> 172.21.15.102:0/3632381742 conn(0x7f5c2c0fcc00 msgr2=0x7f5c2c10cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:06.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 -- 172.21.15.102:0/3632381742 shutdown_connections 2024-04-26T20:35:06.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.696+0000 7f5c33200640 1 -- 172.21.15.102:0/3632381742 wait complete. 2024-04-26T20:35:06.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:35:06.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:06 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2053335074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:35:06.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:06 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2755729068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:35:06.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:06 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3632381742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:35:06.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.923+0000 7f7831c00640 1 -- 172.21.15.102:0/2741861077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f782c108e70 msgr2=0x7f782c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.923+0000 7f7831c00640 1 --2- 172.21.15.102:0/2741861077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f782c108e70 0x7f782c109290 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f781c009860 tx=0x7f781c02a340 comp rx=0 tx=0).stop 2024-04-26T20:35:06.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.923+0000 7f7831c00640 1 -- 172.21.15.102:0/2741861077 shutdown_connections 2024-04-26T20:35:06.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.923+0000 7f7831c00640 1 --2- 172.21.15.102:0/2741861077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f782c109fc0 0x7f782c10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.923+0000 7f7831c00640 1 --2- 172.21.15.102:0/2741861077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f782c10a980 0x7f782c1112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.923+0000 7f7831c00640 1 --2- 172.21.15.102:0/2741861077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f782c108e70 0x7f782c109290 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.923+0000 7f7831c00640 1 -- 172.21.15.102:0/2741861077 >> 172.21.15.102:0/2741861077 conn(0x7f782c077e00 msgr2=0x7f782c07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:06.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.923+0000 7f7831c00640 1 -- 172.21.15.102:0/2741861077 shutdown_connections 2024-04-26T20:35:06.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.923+0000 7f7831c00640 1 -- 172.21.15.102:0/2741861077 wait complete. 2024-04-26T20:35:06.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7831c00640 1 Processor -- start 2024-04-26T20:35:06.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7831c00640 1 -- start start 2024-04-26T20:35:06.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7831c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f782c108e70 0x7f782c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7831c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f782c109fc0 0x7f782c19f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7831c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f782c10a980 0x7f782c1a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7831c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f782c1149c0 con 0x7f782c109fc0 2024-04-26T20:35:06.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7831c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f782c114840 con 0x7f782c108e70 2024-04-26T20:35:06.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7831c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f782c114b40 con 0x7f782c10a980 2024-04-26T20:35:06.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7823e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f782c109fc0 0x7f782c19f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7831200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f782c10a980 0x7f782c1a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7823e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f782c109fc0 0x7f782c19f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43584/0 (socket says 172.21.15.102:43584) 2024-04-26T20:35:06.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f7823e00640 1 -- 172.21.15.102:0/2028411247 learned_addr learned my addr 172.21.15.102:0/2028411247 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:06.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.925+0000 7f782be00640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f782c108e70 0x7f782c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.926+0000 7f7831200640 1 -- 172.21.15.102:0/2028411247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f782c108e70 msgr2=0x7f782c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.926+0000 7f7831200640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f782c108e70 0x7f782c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.926+0000 7f7831200640 1 -- 172.21.15.102:0/2028411247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f782c109fc0 msgr2=0x7f782c19f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.926+0000 7f7831200640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f782c109fc0 0x7f782c19f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.926+0000 7f7831200640 1 -- 172.21.15.102:0/2028411247 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782c1a6c50 con 0x7f782c10a980 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.926+0000 7f7823e00640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f782c109fc0 0x7f782c19f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.926+0000 7f7831200640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f782c10a980 0x7f782c1a65e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7824009fd0 tx=0x7f782400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.926+0000 7f7829600640 1 -- 172.21.15.102:0/2028411247 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782400ce20 con 0x7f782c10a980 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.927+0000 7f7829600640 1 -- 172.21.15.102:0/2028411247 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7824005020 con 0x7f782c10a980 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.927+0000 7f7829600640 1 -- 172.21.15.102:0/2028411247 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824010690 con 0x7f782c10a980 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.927+0000 7f7831c00640 1 -- 172.21.15.102:0/2028411247 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f782c1a6f40 con 0x7f782c10a980 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.927+0000 7f7831c00640 1 -- 172.21.15.102:0/2028411247 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f782c1a73b0 con 0x7f782c10a980 2024-04-26T20:35:06.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.928+0000 7f7831c00640 1 -- 172.21.15.102:0/2028411247 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f782c110180 con 0x7f782c10a980 2024-04-26T20:35:06.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.930+0000 7f7829600640 1 -- 172.21.15.102:0/2028411247 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f78240054d0 con 0x7f782c10a980 2024-04-26T20:35:06.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.931+0000 7f7829600640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f78100a3cc0 0x7f78100a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:06.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.931+0000 7f7829600640 1 -- 172.21.15.102:0/2028411247 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f78240d1410 con 0x7f782c10a980 2024-04-26T20:35:06.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.932+0000 7f782be00640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f78100a3cc0 0x7f78100a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:06.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.932+0000 7f782be00640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f78100a3cc0 0x7f78100a6180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f781c02a930 tx=0x7f781c030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:06.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:06.933+0000 7f7829600640 1 -- 172.21.15.102:0/2028411247 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f782409d880 con 0x7f782c10a980 2024-04-26T20:35:07.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:06 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2053335074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:35:07.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:06 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2755729068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:35:07.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:06 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3632381742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:35:07.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:06 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2053335074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:35:07.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:06 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2755729068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:35:07.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:06 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3632381742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:35:07.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.136+0000 7f7831c00640 1 -- 172.21.15.102:0/2028411247 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f782c02dd20 con 0x7f782c10a980 2024-04-26T20:35:07.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.137+0000 7f7829600640 1 -- 172.21.15.102:0/2028411247 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f782409d180 con 0x7f782c10a980 2024-04-26T20:35:07.138 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:35:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.139+0000 7f7831c00640 1 -- 172.21.15.102:0/2028411247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f78100a3cc0 msgr2=0x7f78100a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.139+0000 7f7831c00640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f78100a3cc0 0x7f78100a6180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f781c02a930 tx=0x7f781c030450 comp rx=0 tx=0).stop 2024-04-26T20:35:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.140+0000 7f7831c00640 1 -- 172.21.15.102:0/2028411247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f782c10a980 msgr2=0x7f782c1a65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.140+0000 7f7831c00640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f782c10a980 0x7f782c1a65e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7824009fd0 tx=0x7f782400ef10 comp rx=0 tx=0).stop 2024-04-26T20:35:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.140+0000 7f7831c00640 1 -- 172.21.15.102:0/2028411247 shutdown_connections 2024-04-26T20:35:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.140+0000 7f7831c00640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f78100a3cc0 0x7f78100a6180 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.140+0000 7f7831c00640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f782c109fc0 0x7f782c19f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.140+0000 7f7831c00640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f782c10a980 0x7f782c1a65e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.140+0000 7f7831c00640 1 --2- 172.21.15.102:0/2028411247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f782c108e70 0x7f782c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.140+0000 7f7831c00640 1 -- 172.21.15.102:0/2028411247 >> 172.21.15.102:0/2028411247 conn(0x7f782c077e00 msgr2=0x7f782c10d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:07.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.140+0000 7f7831c00640 1 -- 172.21.15.102:0/2028411247 shutdown_connections 2024-04-26T20:35:07.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.140+0000 7f7831c00640 1 -- 172.21.15.102:0/2028411247 wait complete. 2024-04-26T20:35:07.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:35:07.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.360+0000 7f841e000640 1 -- 172.21.15.102:0/72820492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8418076910 msgr2=0x7f8418074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:07.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.360+0000 7f841e000640 1 --2- 172.21.15.102:0/72820492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8418076910 0x7f8418074d60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8400009900 tx=0x7f840002a480 comp rx=0 tx=0).stop 2024-04-26T20:35:07.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.360+0000 7f841e000640 1 -- 172.21.15.102:0/72820492 shutdown_connections 2024-04-26T20:35:07.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.360+0000 7f841e000640 1 --2- 172.21.15.102:0/72820492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8418075330 0x7f8418075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.360+0000 7f841e000640 1 --2- 172.21.15.102:0/72820492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f841810a810 0x7f841810ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.360+0000 7f841e000640 1 --2- 172.21.15.102:0/72820492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8418076910 0x7f8418074d60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.360+0000 7f841e000640 1 -- 172.21.15.102:0/72820492 >> 172.21.15.102:0/72820492 conn(0x7f84180fdf10 msgr2=0x7f8418100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:07.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.361+0000 7f841e000640 1 -- 172.21.15.102:0/72820492 shutdown_connections 2024-04-26T20:35:07.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.361+0000 7f841e000640 1 -- 172.21.15.102:0/72820492 wait complete. 2024-04-26T20:35:07.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.361+0000 7f841e000640 1 Processor -- start 2024-04-26T20:35:07.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f841e000640 1 -- start start 2024-04-26T20:35:07.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f841e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8418075330 0x7f841819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:07.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f841e000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8418076910 0x7f841819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:07.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f841e000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f841810a810 0x7f84181a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:07.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f841e000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f84181106f0 con 0x7f8418075330 2024-04-26T20:35:07.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f841e000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8418110570 con 0x7f841810a810 2024-04-26T20:35:07.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f841e000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8418110870 con 0x7f8418076910 2024-04-26T20:35:07.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f8417400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8418075330 0x7f841819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f8417400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8418075330 0x7f841819adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43598/0 (socket says 172.21.15.102:43598) 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f8417400640 1 -- 172.21.15.102:0/3788177515 learned_addr learned my addr 172.21.15.102:0/3788177515 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f8416a00640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8418076910 0x7f841819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.362+0000 7f8417e00640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f841810a810 0x7f84181a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.363+0000 7f8417400640 1 -- 172.21.15.102:0/3788177515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8418076910 msgr2=0x7f841819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.363+0000 7f8417400640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8418076910 0x7f841819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.363+0000 7f8417400640 1 -- 172.21.15.102:0/3788177515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f841810a810 msgr2=0x7f84181a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.363+0000 7f8417400640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f841810a810 0x7f84181a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.363+0000 7f8417400640 1 -- 172.21.15.102:0/3788177515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84181a29c0 con 0x7f8418075330 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.363+0000 7f8416a00640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8418076910 0x7f841819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:07.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.363+0000 7f8417400640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8418075330 0x7f841819adb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8400005ab0 tx=0x7f8400003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:07.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.363+0000 7f840fe00640 1 -- 172.21.15.102:0/3788177515 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840003f070 con 0x7f8418075330 2024-04-26T20:35:07.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.364+0000 7f841e000640 1 -- 172.21.15.102:0/3788177515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84181a2c50 con 0x7f8418075330 2024-04-26T20:35:07.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.364+0000 7f840fe00640 1 -- 172.21.15.102:0/3788177515 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8400031c50 con 0x7f8418075330 2024-04-26T20:35:07.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.364+0000 7f840fe00640 1 -- 172.21.15.102:0/3788177515 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400035030 con 0x7f8418075330 2024-04-26T20:35:07.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.364+0000 7f841e000640 1 -- 172.21.15.102:0/3788177515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84181a3140 con 0x7f8418075330 2024-04-26T20:35:07.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.365+0000 7f841e000640 1 -- 172.21.15.102:0/3788177515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84181095d0 con 0x7f8418075330 2024-04-26T20:35:07.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.368+0000 7f840fe00640 1 -- 172.21.15.102:0/3788177515 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8400042070 con 0x7f8418075330 2024-04-26T20:35:07.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.370+0000 7f840fe00640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3c40 0x7f83fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:07.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.370+0000 7f840fe00640 1 -- 172.21.15.102:0/3788177515 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f84000f17b0 con 0x7f8418075330 2024-04-26T20:35:07.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.370+0000 7f8416a00640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3c40 0x7f83fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:07.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.371+0000 7f8416a00640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3c40 0x7f83fc0a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8408009fd0 tx=0x7f8408009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:07.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.372+0000 7f840fe00640 1 -- 172.21.15.102:0/3788177515 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f84000bd630 con 0x7f8418075330 2024-04-26T20:35:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.615+0000 7f841e000640 1 -- 172.21.15.102:0/3788177515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f841802dad0 con 0x7f8418075330 2024-04-26T20:35:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.617+0000 7f840fe00640 1 -- 172.21.15.102:0/3788177515 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11954 (secure 0 0 0) 0x7f84000c1930 con 0x7f8418075330 2024-04-26T20:35:07.618 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:35:07.618 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:32:25.769708+0000","modified":"2024-04-26T20:32:55.699799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24593,"mds_1":24565,"mds_2":24547,"mds_3":24533,"mds_4":24526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24565,24593,24547,24533,24526]},"id":1}]} 2024-04-26T20:35:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-26T20:35:07.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.619+0000 7f841e000640 1 -- 172.21.15.102:0/3788177515 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3c40 msgr2=0x7f83fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:07.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.619+0000 7f841e000640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3c40 0x7f83fc0a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8408009fd0 tx=0x7f8408009680 comp rx=0 tx=0).stop 2024-04-26T20:35:07.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.619+0000 7f841e000640 1 -- 172.21.15.102:0/3788177515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8418075330 msgr2=0x7f841819adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:07.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.619+0000 7f841e000640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8418075330 0x7f841819adb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8400005ab0 tx=0x7f8400003b90 comp rx=0 tx=0).stop 2024-04-26T20:35:07.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.619+0000 7f841e000640 1 -- 172.21.15.102:0/3788177515 shutdown_connections 2024-04-26T20:35:07.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.619+0000 7f841e000640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f83fc0a3c40 0x7f83fc0a6100 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.619+0000 7f841e000640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f841810a810 0x7f84181a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.619+0000 7f841e000640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8418076910 0x7f841819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.619+0000 7f841e000640 1 --2- 172.21.15.102:0/3788177515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8418075330 0x7f841819adb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.619+0000 7f841e000640 1 -- 172.21.15.102:0/3788177515 >> 172.21.15.102:0/3788177515 conn(0x7f84180fdf10 msgr2=0x7f84181067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:07.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.620+0000 7f841e000640 1 -- 172.21.15.102:0/3788177515 shutdown_connections 2024-04-26T20:35:07.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.620+0000 7f841e000640 1 -- 172.21.15.102:0/3788177515 wait complete. 2024-04-26T20:35:07.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:35:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.837+0000 7f3253200640 1 -- 172.21.15.102:0/1320407344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3254105ba0 msgr2=0x7f3254106020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.837+0000 7f3253200640 1 --2- 172.21.15.102:0/1320407344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3254105ba0 0x7f3254106020 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f323c009900 tx=0x7f323c02a410 comp rx=0 tx=0).stop 2024-04-26T20:35:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.837+0000 7f3253200640 1 -- 172.21.15.102:0/1320407344 shutdown_connections 2024-04-26T20:35:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.837+0000 7f3253200640 1 --2- 172.21.15.102:0/1320407344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3254106560 0x7f325410cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.837+0000 7f3253200640 1 --2- 172.21.15.102:0/1320407344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3254105ba0 0x7f3254106020 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.837+0000 7f3253200640 1 --2- 172.21.15.102:0/1320407344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32541049b0 0x7f3254104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.837+0000 7f3253200640 1 -- 172.21.15.102:0/1320407344 >> 172.21.15.102:0/1320407344 conn(0x7f32540fff40 msgr2=0x7f3254102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:07.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.837+0000 7f3253200640 1 -- 172.21.15.102:0/1320407344 shutdown_connections 2024-04-26T20:35:07.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.837+0000 7f3253200640 1 -- 172.21.15.102:0/1320407344 wait complete. 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.838+0000 7f3253200640 1 Processor -- start 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.838+0000 7f3253200640 1 -- start start 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f3253200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32541049b0 0x7f325419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f3253200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3254105ba0 0x7f325419b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f3253200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3254106560 0x7f32541a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f3253200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3254110530 con 0x7f32541049b0 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f3253200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f32541103b0 con 0x7f3254105ba0 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f3253200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f32541106b0 con 0x7f3254106560 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f3251e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32541049b0 0x7f325419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f324be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3254105ba0 0x7f325419b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f324be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3254105ba0 0x7f325419b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:60332/0 (socket says 172.21.15.102:60332) 2024-04-26T20:35:07.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f324be00640 1 -- 172.21.15.102:0/2982538165 learned_addr learned my addr 172.21.15.102:0/2982538165 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:07.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f3251e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32541049b0 0x7f325419ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43618/0 (socket says 172.21.15.102:43618) 2024-04-26T20:35:07.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f3252800640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3254106560 0x7f32541a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:07.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.839+0000 7f3251e00640 1 -- 172.21.15.102:0/2982538165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3254106560 msgr2=0x7f32541a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:07.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f3251e00640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3254106560 0x7f32541a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f3251e00640 1 -- 172.21.15.102:0/2982538165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3254105ba0 msgr2=0x7f325419b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:07.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f3251e00640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3254105ba0 0x7f325419b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:07.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f3251e00640 1 -- 172.21.15.102:0/2982538165 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32541a27a0 con 0x7f32541049b0 2024-04-26T20:35:07.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f324be00640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3254105ba0 0x7f325419b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:07.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f3251e00640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32541049b0 0x7f325419ab80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f324400e9c0 tx=0x7f324400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:07.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f324aa00640 1 -- 172.21.15.102:0/2982538165 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324400ccc0 con 0x7f32541049b0 2024-04-26T20:35:07.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f324aa00640 1 -- 172.21.15.102:0/2982538165 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32440035f0 con 0x7f32541049b0 2024-04-26T20:35:07.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f3253200640 1 -- 172.21.15.102:0/2982538165 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32541a2a90 con 0x7f32541049b0 2024-04-26T20:35:07.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f324aa00640 1 -- 172.21.15.102:0/2982538165 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324400ce60 con 0x7f32541049b0 2024-04-26T20:35:07.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.840+0000 7f3253200640 1 -- 172.21.15.102:0/2982538165 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32541a2f90 con 0x7f32541049b0 2024-04-26T20:35:07.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.841+0000 7f3253200640 1 -- 172.21.15.102:0/2982538165 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f325410bcd0 con 0x7f32541049b0 2024-04-26T20:35:07.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.843+0000 7f324aa00640 1 -- 172.21.15.102:0/2982538165 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f3244004000 con 0x7f32541049b0 2024-04-26T20:35:07.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.845+0000 7f324aa00640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f32380a3c40 0x7f32380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:07.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.845+0000 7f324aa00640 1 -- 172.21.15.102:0/2982538165 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f32440d0d10 con 0x7f32541049b0 2024-04-26T20:35:07.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.845+0000 7f324be00640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f32380a3c40 0x7f32380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:07.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.846+0000 7f324be00640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f32380a3c40 0x7f32380a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f325419c090 tx=0x7f323c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:07.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:07.849+0000 7f324aa00640 1 -- 172.21.15.102:0/2982538165 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f32440d6020 con 0x7f32541049b0 2024-04-26T20:35:07.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:07 smithi102 ceph-mon[28094]: pgmap v402: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:35:07.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:07 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2028411247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:35:07.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:07 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3788177515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:35:08.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:07 smithi112 ceph-mon[32769]: pgmap v402: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:35:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:07 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2028411247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:35:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:07 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3788177515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:35:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.055+0000 7f3253200640 1 -- 172.21.15.102:0/2982538165 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f325402dad0 con 0x7f32541049b0 2024-04-26T20:35:08.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.059+0000 7f324aa00640 1 -- 172.21.15.102:0/2982538165 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24833 (secure 0 0 0) 0x7f324409d180 con 0x7f32541049b0 2024-04-26T20:35:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:35:08.061 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:34:30.697401+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:35:08.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 -- 172.21.15.102:0/2982538165 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f32380a3c40 msgr2=0x7f32380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f32380a3c40 0x7f32380a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f325419c090 tx=0x7f323c005a20 comp rx=0 tx=0).stop 2024-04-26T20:35:08.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 -- 172.21.15.102:0/2982538165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32541049b0 msgr2=0x7f325419ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32541049b0 0x7f325419ab80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f324400e9c0 tx=0x7f324400ee80 comp rx=0 tx=0).stop 2024-04-26T20:35:08.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 -- 172.21.15.102:0/2982538165 shutdown_connections 2024-04-26T20:35:08.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f32380a3c40 0x7f32380a6100 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3254106560 0x7f32541a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3254105ba0 0x7f325419b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 --2- 172.21.15.102:0/2982538165 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32541049b0 0x7f325419ab80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 -- 172.21.15.102:0/2982538165 >> 172.21.15.102:0/2982538165 conn(0x7f32540fff40 msgr2=0x7f3254108ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 -- 172.21.15.102:0/2982538165 shutdown_connections 2024-04-26T20:35:08.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.062+0000 7f3253200640 1 -- 172.21.15.102:0/2982538165 wait complete. 2024-04-26T20:35:08.084 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T20:32:25.769708+0000', 'modified': '2024-04-26T20:32:55.699799+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24593, 'mds_1': 24565, 'mds_2': 24547, 'mds_3': 24533, 'mds_4': 24526}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24448': {'gid': 24448, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6833/4251888549', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6832', 'nonce': 4251888549}, {'type': 'v1', 'addr': '172.21.15.130:6833', 'nonce': 4251888549}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6833/1543441778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6832', 'nonce': 1543441778}, {'type': 'v1', 'addr': '172.21.15.112:6833', 'nonce': 1543441778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'd', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.102:6837/3283929332', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 3283929332}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 3283929332}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6835/2010209065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6834', 'nonce': 2010209065}, {'type': 'v1', 'addr': '172.21.15.112:6835', 'nonce': 2010209065}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24514': {'gid': 24514, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6835/829567775', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 829567775}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 829567775}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24526': {'gid': 24526, 'name': 'c', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.130:6837/2107305806', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 2107305806}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 2107305806}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.112:6837/82964069', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6836', 'nonce': 82964069}, {'type': 'v1', 'addr': '172.21.15.112:6837', 'nonce': 82964069}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.130:6839/1700008514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 1700008514}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 1700008514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24565': {'gid': 24565, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.102:6839/3473487681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6838', 'nonce': 3473487681}, {'type': 'v1', 'addr': '172.21.15.102:6839', 'nonce': 3473487681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24593': {'gid': 24593, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.112:6839/2318597180', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 2318597180}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 2318597180}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24593, 'qdb_cluster': [24565, 24593, 24547, 24533, 24526]} 2024-04-26T20:35:08.084 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-26T20:35:08.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-26T20:35:08.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:07 smithi130 ceph-mon[32704]: pgmap v402: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:35:08.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:07 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2028411247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:35:08.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:07 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3788177515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:35:08.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.284+0000 7f68b1600640 1 -- 172.21.15.102:0/4235605875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68ac108fb0 msgr2=0x7f68ac1093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.284+0000 7f68b1600640 1 --2- 172.21.15.102:0/4235605875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68ac108fb0 0x7f68ac1093d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f689c009900 tx=0x7f689c02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.284+0000 7f68b1600640 1 -- 172.21.15.102:0/4235605875 shutdown_connections 2024-04-26T20:35:08.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.284+0000 7f68b1600640 1 --2- 172.21.15.102:0/4235605875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac10ab60 0x7f68ac1113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.284+0000 7f68b1600640 1 --2- 172.21.15.102:0/4235605875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68ac10a1a0 0x7f68ac10a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.284+0000 7f68b1600640 1 --2- 172.21.15.102:0/4235605875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68ac108fb0 0x7f68ac1093d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.284+0000 7f68b1600640 1 -- 172.21.15.102:0/4235605875 >> 172.21.15.102:0/4235605875 conn(0x7f68ac077f40 msgr2=0x7f68ac07a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.284+0000 7f68b1600640 1 -- 172.21.15.102:0/4235605875 shutdown_connections 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.284+0000 7f68b1600640 1 -- 172.21.15.102:0/4235605875 wait complete. 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.285+0000 7f68b1600640 1 Processor -- start 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.285+0000 7f68b1600640 1 -- start start 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68b1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac108fb0 0x7f68ac19f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68b1600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68ac10a1a0 0x7f68ac19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68b1600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68ac10ab60 0x7f68ac1a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68b1600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac114ab0 con 0x7f68ac108fb0 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68b1600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac114930 con 0x7f68ac10a1a0 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68b1600640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac114c30 con 0x7f68ac10ab60 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac108fb0 0x7f68ac19f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac108fb0 0x7f68ac19f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43646/0 (socket says 172.21.15.102:43646) 2024-04-26T20:35:08.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68aaa00640 1 -- 172.21.15.102:0/3395844635 learned_addr learned my addr 172.21.15.102:0/3395844635 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68ab400640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68ac10ab60 0x7f68ac1a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.286+0000 7f68aa000640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68ac10a1a0 0x7f68ac19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68aaa00640 1 -- 172.21.15.102:0/3395844635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68ac10ab60 msgr2=0x7f68ac1a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68aaa00640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68ac10ab60 0x7f68ac1a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68aaa00640 1 -- 172.21.15.102:0/3395844635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68ac10a1a0 msgr2=0x7f68ac19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68aaa00640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68ac10a1a0 0x7f68ac19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68aaa00640 1 -- 172.21.15.102:0/3395844635 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68ac1a6dd0 con 0x7f68ac108fb0 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68ab400640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68ac10ab60 0x7f68ac1a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68aa000640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68ac10a1a0 0x7f68ac19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68aaa00640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac108fb0 0x7f68ac19f1b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f689c006010 tx=0x7f689c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68a3400640 1 -- 172.21.15.102:0/3395844635 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c03f030 con 0x7f68ac108fb0 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68b1600640 1 -- 172.21.15.102:0/3395844635 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68ac1a7060 con 0x7f68ac108fb0 2024-04-26T20:35:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68a3400640 1 -- 172.21.15.102:0/3395844635 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f689c031910 con 0x7f68ac108fb0 2024-04-26T20:35:08.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68a3400640 1 -- 172.21.15.102:0/3395844635 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c035070 con 0x7f68ac108fb0 2024-04-26T20:35:08.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.287+0000 7f68b1600640 1 -- 172.21.15.102:0/3395844635 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68ac1a75a0 con 0x7f68ac108fb0 2024-04-26T20:35:08.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.290+0000 7f68b444c740 1 Processor -- start 2024-04-26T20:35:08.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68b444c740 1 -- start start 2024-04-26T20:35:08.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68a3400640 1 -- 172.21.15.102:0/3395844635 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f689c042030 con 0x7f68ac108fb0 2024-04-26T20:35:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68b444c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189e0f0 0x56079189e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68b444c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189eae0 0x5607918a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68b444c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607918a3c50 0x5607918a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68b444c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560791802600 con 0x56079189e0f0 2024-04-26T20:35:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68b444c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56079179f100 con 0x56079189eae0 2024-04-26T20:35:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68b444c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5607917fe5c0 con 0x5607918a3c50 2024-04-26T20:35:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68aaa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189eae0 0x5607918a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68ab400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189e0f0 0x56079189e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68aaa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189eae0 0x5607918a35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:60360/0 (socket says 172.21.15.102:60360) 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.291+0000 7f68aaa00640 1 -- 172.21.15.102:0/690206322 learned_addr learned my addr 172.21.15.102:0/690206322 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.292+0000 7f68aa000640 1 --2- 172.21.15.102:0/690206322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607918a3c50 0x5607918a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.292+0000 7f68aaa00640 1 -- 172.21.15.102:0/690206322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607918a3c50 msgr2=0x5607918a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.292+0000 7f68aaa00640 1 --2- 172.21.15.102:0/690206322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607918a3c50 0x5607918a60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.292+0000 7f68aaa00640 1 -- 172.21.15.102:0/690206322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189e0f0 msgr2=0x56079189e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.292+0000 7f68aaa00640 1 --2- 172.21.15.102:0/690206322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189e0f0 0x56079189e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.292+0000 7f68aaa00640 1 -- 172.21.15.102:0/690206322 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607915142d0 con 0x56079189eae0 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.292+0000 7f68ab400640 1 --2- 172.21.15.102:0/690206322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189e0f0 0x56079189e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.292+0000 7f68aaa00640 1 --2- 172.21.15.102:0/690206322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189eae0 0x5607918a35e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f689c0ee370 tx=0x7f689c0f91a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e907db09615eee6a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.293+0000 7f6897400640 1 -- 172.21.15.102:0/690206322 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c0f9de0 con 0x56079189eae0 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.293+0000 7f68a3400640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68900a3c40 0x7f68900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.293+0000 7f6897400640 1 -- 172.21.15.102:0/690206322 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f689c0fe070 con 0x56079189eae0 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.293+0000 7f6897400640 1 -- 172.21.15.102:0/690206322 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c0b8f50 con 0x56079189eae0 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.293+0000 7f68a3400640 1 -- 172.21.15.102:0/3395844635 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f689c0f0e30 con 0x7f68ac108fb0 2024-04-26T20:35:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.293+0000 7f68ab400640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68900a3c40 0x7f68900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.293+0000 7f68b444c740 1 -- 172.21.15.102:0/690206322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189eae0 msgr2=0x5607918a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.293+0000 7f68b444c740 1 --2- 172.21.15.102:0/690206322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189eae0 0x5607918a35e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f689c0ee370 tx=0x7f689c0f91a0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.294+0000 7f68ab400640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68900a3c40 0x7f68900a6100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x56079189f340 tx=0x7f68a4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.294+0000 7f68b444c740 1 -- 172.21.15.102:0/690206322 shutdown_connections 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.294+0000 7f68b444c740 1 --2- 172.21.15.102:0/690206322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607918a3c50 0x5607918a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.294+0000 7f68b444c740 1 --2- 172.21.15.102:0/690206322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189eae0 0x5607918a35e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.294+0000 7f68b444c740 1 --2- 172.21.15.102:0/690206322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189e0f0 0x56079189e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.294+0000 7f68b444c740 1 -- 172.21.15.102:0/690206322 >> 172.21.15.102:0/690206322 conn(0x560791832d10 msgr2=0x560791899b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.294+0000 7f68b444c740 1 -- 172.21.15.102:0/690206322 shutdown_connections 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.294+0000 7f68b444c740 1 -- 172.21.15.102:0/690206322 wait complete. 2024-04-26T20:35:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.294+0000 7f68b444c740 1 -- start start 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.294+0000 7f68b444c740 1 Processor -- start 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68b444c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x5607918a8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68b444c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68b444c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68b444c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5607918a69c0 con 0x5607918a3c50 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68b444c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560791789f70 con 0x56079189e0f0 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68b444c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5607918a6b40 con 0x56079189eae0 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68b444c740 10 client.0 _ll_register_callbacks cb 0x5607917f8480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68ab400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68ab400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918af7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43666/0 (socket says 172.21.15.102:43666) 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68ab400640 1 -- 172.21.15.102:0/3779153821 learned_addr learned my addr 172.21.15.102:0/3779153821 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68aa000640 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.295+0000 7f68aaa00640 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x5607918a8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f68ab400640 1 -- 172.21.15.102:0/3779153821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 msgr2=0x5607918a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f68ab400640 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918a8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f68ab400640 1 -- 172.21.15.102:0/3779153821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 msgr2=0x5607918a8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f68ab400640 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x5607918a8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f68ab400640 1 -- 172.21.15.102:0/3779153821 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607915de6b0 con 0x5607918a3c50 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f68aa000640 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f68aaa00640 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x5607918a8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f68ab400640 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918af7b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f68a40116f0 tx=0x7f68a4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f706a30fcd823d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f688be00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f688be00640 1 -- 172.21.15.102:0/3779153821 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a402c060 con 0x5607918a3c50 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f688be00640 1 -- 172.21.15.102:0/3779153821 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f68a4017030 con 0x5607918a3c50 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.296+0000 7f68b444c740 1 -- 172.21.15.102:0/3779153821 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f689c0fe070 con 0x5607918a3c50 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f688be00640 1 -- 172.21.15.102:0/3779153821 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f68a4033050 con 0x5607918a3c50 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f68b444c740 10 client.15246 fetch_fsmap learned FSMap version 27 2024-04-26T20:35:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f68b444c740 1 -- 172.21.15.102:0/3779153821 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607918b2860 con 0x5607918a3c50 2024-04-26T20:35:08.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f688be00640 1 -- 172.21.15.102:0/3779153821 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f689c0fe070 con 0x5607918a3c50 2024-04-26T20:35:08.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f688be00640 1 -- 172.21.15.102:0/3779153821 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f68a4010a70 con 0x5607918a3c50 2024-04-26T20:35:08.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f68b444c740 10 client.15246 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:35:08.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f68b444c740 10 client.15246 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T20:35:08.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f68b444c740 20 client.15246 populate_metadata read hostname 'smithi102' 2024-04-26T20:35:08.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f68b444c740 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x5607918b4f40 0x5607918b63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f68b444c740 4 client.15246 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:35:08.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.297+0000 7f68b444c740 1 -- 172.21.15.102:0/3779153821 --> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560791810210 con 0x5607918b4f40 2024-04-26T20:35:08.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.298+0000 7f68aaa00640 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x5607918b4f40 0x5607918b63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.299+0000 7f688be00640 10 client.15246 ms_handle_connect on v2:172.21.15.130:6836/2107305806 2024-04-26T20:35:08.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.299+0000 7f68aaa00640 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x5607918b4f40 0x5607918b63b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b80e1c49ec90d560 server_cookie=6cad59849eec6585 in_seq=0 out_seq=0 2024-04-26T20:35:08.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.299+0000 7f688be00640 1 -- 172.21.15.102:0/3779153821 <== mds.4 v2:172.21.15.130:6836/2107305806 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560791810210 con 0x5607918b4f40 2024-04-26T20:35:08.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.299+0000 7f688be00640 10 client.15246 handle_command_reply: tid=0 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 client.15246 shutdown 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 -- 172.21.15.102:0/3779153821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 msgr2=0x5607918af7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918af7b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f68a40116f0 tx=0x7f68a4004ab0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 -- 172.21.15.102:0/3779153821 shutdown_connections 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x5607918b4f40 0x5607918b63b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918af7b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f688be00640 0 client.15246 ms_handle_reset on v2:172.21.15.130:6836/2107305806 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 --2- 172.21.15.102:0/3779153821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x5607918a8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 -- 172.21.15.102:0/3779153821 >> 172.21.15.102:0/3779153821 conn(0x560791832d10 msgr2=0x560791898eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 -- 172.21.15.102:0/3779153821 shutdown_connections 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 1 -- 172.21.15.102:0/3779153821 wait complete. 2024-04-26T20:35:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.300+0000 7f68b444c740 20 client.15246 trim_cache size 0 max 16384 2024-04-26T20:35:08.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.319+0000 7f68b444c740 1 Processor -- start 2024-04-26T20:35:08.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.319+0000 7f68b444c740 1 -- start start 2024-04-26T20:35:08.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68b444c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x56079189fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68b444c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189eae0 0x5607918af770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68b444c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607918a3c50 0x5607918a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68b444c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5607918b46f0 con 0x56079189eae0 2024-04-26T20:35:08.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68b444c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5607918b4dc0 con 0x56079189e0f0 2024-04-26T20:35:08.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68b444c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5607918b5490 con 0x5607918a3c50 2024-04-26T20:35:08.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68aa000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189eae0 0x5607918af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68aa000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189eae0 0x5607918af770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43682/0 (socket says 172.21.15.102:43682) 2024-04-26T20:35:08.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68aaa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x56079189fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68aa000640 1 -- 172.21.15.102:0/3349899273 learned_addr learned my addr 172.21.15.102:0/3349899273 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68ab400640 1 --2- 172.21.15.102:0/3349899273 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607918a3c50 0x5607918a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68aaa00640 1 -- 172.21.15.102:0/3349899273 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607918a3c50 msgr2=0x5607918a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68aaa00640 1 --2- 172.21.15.102:0/3349899273 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607918a3c50 0x5607918a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68aaa00640 1 -- 172.21.15.102:0/3349899273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189eae0 msgr2=0x5607918af770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68aaa00640 1 --2- 172.21.15.102:0/3349899273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189eae0 0x5607918af770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68aaa00640 1 -- 172.21.15.102:0/3349899273 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607918958a0 con 0x56079189e0f0 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.320+0000 7f68aa000640 1 --2- 172.21.15.102:0/3349899273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189eae0 0x5607918af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f68aaa00640 1 --2- 172.21.15.102:0/3349899273 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x56079189fea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f689c0c4020 tx=0x7f689c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69417a5eee8db0ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f6896a00640 1 -- 172.21.15.102:0/3349899273 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c0bc430 con 0x56079189e0f0 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f6896a00640 1 -- 172.21.15.102:0/3349899273 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f689c0fe040 con 0x56079189e0f0 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f6896a00640 1 -- 172.21.15.102:0/3349899273 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c0bc610 con 0x56079189e0f0 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f68b444c740 1 -- 172.21.15.102:0/3349899273 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 msgr2=0x56079189fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f68b444c740 1 --2- 172.21.15.102:0/3349899273 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x56079189fea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f689c0c4020 tx=0x7f689c009260 comp rx=0 tx=0).stop 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f68b444c740 1 -- 172.21.15.102:0/3349899273 shutdown_connections 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f68b444c740 1 --2- 172.21.15.102:0/3349899273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56079189eae0 0x5607918af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f68b444c740 1 --2- 172.21.15.102:0/3349899273 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607918a3c50 0x5607918a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f68b444c740 1 --2- 172.21.15.102:0/3349899273 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x56079189fea0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.321+0000 7f68b444c740 1 -- 172.21.15.102:0/3349899273 >> 172.21.15.102:0/3349899273 conn(0x560791832d10 msgr2=0x5607918a81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.322+0000 7f68b444c740 1 -- 172.21.15.102:0/3349899273 shutdown_connections 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.322+0000 7f68b444c740 1 -- 172.21.15.102:0/3349899273 wait complete. 2024-04-26T20:35:08.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.322+0000 7f68b444c740 1 -- start start 2024-04-26T20:35:08.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.322+0000 7f68b444c740 1 Processor -- start 2024-04-26T20:35:08.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.322+0000 7f68b444c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x5607918aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.322+0000 7f68b444c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.322+0000 7f68b444c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.322+0000 7f68b444c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5607918b6230 con 0x5607918a3c50 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68b444c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5607918b5b60 con 0x56079189e0f0 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68b444c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5607918b6900 con 0x56079189eae0 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68b444c740 10 client.0 _ll_register_callbacks cb 0x7f68a4003c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68aa000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68aa000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918ab1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:34938/0 (socket says 172.21.15.102:34938) 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68aa000640 1 -- 172.21.15.102:0/4166159836 learned_addr learned my addr 172.21.15.102:0/4166159836 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68ab400640 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68aaa00640 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x5607918aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68aa000640 1 -- 172.21.15.102:0/4166159836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 msgr2=0x5607918aacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68aa000640 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x5607918aacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68aa000640 1 -- 172.21.15.102:0/4166159836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 msgr2=0x5607918ae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68aa000640 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918ae1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68aa000640 1 -- 172.21.15.102:0/4166159836 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607918ae790 con 0x56079189eae0 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.323+0000 7f68ab400640 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f68aa000640 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918ab1f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f689800f410 tx=0x7f689800f8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77f20a7c07563f1f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f688be00640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f688be00640 1 -- 172.21.15.102:0/4166159836 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898021030 con 0x56079189eae0 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f68b444c740 1 -- 172.21.15.102:0/4166159836 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f689c0fe040 con 0x56079189eae0 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f688be00640 1 -- 172.21.15.102:0/4166159836 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f6898004020 con 0x56079189eae0 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f688be00640 1 -- 172.21.15.102:0/4166159836 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f689802b050 con 0x56079189eae0 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f68b444c740 10 client.25372 fetch_fsmap learned FSMap version 27 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f68b444c740 1 -- 172.21.15.102:0/4166159836 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56079191aef0 con 0x56079189eae0 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f688be00640 1 -- 172.21.15.102:0/4166159836 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f689c0fe040 con 0x56079189eae0 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f688be00640 1 -- 172.21.15.102:0/4166159836 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6898026450 con 0x56079189eae0 2024-04-26T20:35:08.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.324+0000 7f68b444c740 10 client.25372 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.325+0000 7f68b444c740 10 client.25372 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.325+0000 7f68b444c740 20 client.25372 populate_metadata read hostname 'smithi102' 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.325+0000 7f68b444c740 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x56079191fb10 0x56079191e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.325+0000 7f68b444c740 4 client.25372 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.325+0000 7f68b444c740 1 -- 172.21.15.102:0/4166159836 --> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] -- command(tid 0: {"prefix": "status"}) -- 0x56079180a080 con 0x56079191fb10 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.325+0000 7f68aaa00640 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x56079191fb10 0x56079191e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.325+0000 7f68aaa00640 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x56079191fb10 0x56079191e820 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1cb398ea332a9261 server_cookie=c8b68d256f025bb6 in_seq=0 out_seq=0 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.325+0000 7f688be00640 10 client.25372 ms_handle_connect on v2:172.21.15.130:6836/2107305806 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f688be00640 1 -- 172.21.15.102:0/4166159836 <== mds.4 v2:172.21.15.130:6836/2107305806 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56079180a080 con 0x56079191fb10 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f688be00640 10 client.25372 handle_command_reply: tid=0 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 client.25372 shutdown 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 -- 172.21.15.102:0/4166159836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 msgr2=0x5607918ab1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918ab1f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f689800f410 tx=0x7f689800f8d0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 -- 172.21.15.102:0/4166159836 shutdown_connections 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.130:6836/2107305806,v1:172.21.15.130:6837/2107305806] conn(0x56079191fb10 0x56079191e820 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607918a3c50 0x5607918ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f688be00640 0 client.25372 ms_handle_reset on v2:172.21.15.130:6836/2107305806 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56079189eae0 0x5607918ab1f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 --2- 172.21.15.102:0/4166159836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56079189e0f0 0x5607918aacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 -- 172.21.15.102:0/4166159836 >> 172.21.15.102:0/4166159836 conn(0x560791832d10 msgr2=0x5607918a81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 -- 172.21.15.102:0/4166159836 shutdown_connections 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 1 -- 172.21.15.102:0/4166159836 wait complete. 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.326+0000 7f68b444c740 20 client.25372 trim_cache size 0 max 16384 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:35:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 4, 2024-04-26T20:35:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24526, 2024-04-26T20:35:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:35:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:35:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:35:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 133.65156585599999, 2024-04-26T20:35:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 27, 2024-04-26T20:35:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-04-26T20:35:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:35:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 185.41778379300001 2024-04-26T20:35:08.333 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 -- 172.21.15.102:0/3395844635 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68900a3c40 msgr2=0x7f68900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68900a3c40 0x7f68900a6100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x56079189f340 tx=0x7f68a4009500 comp rx=0 tx=0).stop 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 -- 172.21.15.102:0/3395844635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac108fb0 msgr2=0x7f68ac19f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac108fb0 0x7f68ac19f1b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f689c006010 tx=0x7f689c0042f0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 -- 172.21.15.102:0/3395844635 shutdown_connections 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f68900a3c40 0x7f68900a6100 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68ac10ab60 0x7f68ac1a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68ac10a1a0 0x7f68ac19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 --2- 172.21.15.102:0/3395844635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac108fb0 0x7f68ac19f1b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 -- 172.21.15.102:0/3395844635 >> 172.21.15.102:0/3395844635 conn(0x7f68ac077f40 msgr2=0x7f68ac10d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 -- 172.21.15.102:0/3395844635 shutdown_connections 2024-04-26T20:35:08.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.327+0000 7f68b1600640 1 -- 172.21.15.102:0/3395844635 wait complete. 2024-04-26T20:35:08.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-26T20:35:08.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.544+0000 7f4293e00640 1 -- 172.21.15.102:0/1426552488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4294100a20 msgr2=0x7f4294100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.544+0000 7f4293e00640 1 --2- 172.21.15.102:0/1426552488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4294100a20 0x7f4294100e40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4284009930 tx=0x7f428402a440 comp rx=0 tx=0).stop 2024-04-26T20:35:08.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.544+0000 7f4293e00640 1 -- 172.21.15.102:0/1426552488 shutdown_connections 2024-04-26T20:35:08.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.544+0000 7f4293e00640 1 --2- 172.21.15.102:0/1426552488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42941043b0 0x7f429410ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.544+0000 7f4293e00640 1 --2- 172.21.15.102:0/1426552488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42941039f0 0x7f4294103e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.544+0000 7f4293e00640 1 --2- 172.21.15.102:0/1426552488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4294100a20 0x7f4294100e40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.544+0000 7f4293e00640 1 -- 172.21.15.102:0/1426552488 >> 172.21.15.102:0/1426552488 conn(0x7f42940fc350 msgr2=0x7f42940fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.544+0000 7f4293e00640 1 -- 172.21.15.102:0/1426552488 shutdown_connections 2024-04-26T20:35:08.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.544+0000 7f4293e00640 1 -- 172.21.15.102:0/1426552488 wait complete. 2024-04-26T20:35:08.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.545+0000 7f4293e00640 1 Processor -- start 2024-04-26T20:35:08.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.545+0000 7f4293e00640 1 -- start start 2024-04-26T20:35:08.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.545+0000 7f4293e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4294100a20 0x7f429419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.545+0000 7f4293e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42941039f0 0x7f429419b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.545+0000 7f4293e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42941043b0 0x7f42941a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f4293e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f429410e2d0 con 0x7f42941039f0 2024-04-26T20:35:08.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f4293e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f429410e150 con 0x7f4294100a20 2024-04-26T20:35:08.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f4293e00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f429410e450 con 0x7f42941043b0 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f428be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42941039f0 0x7f429419b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f428be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42941039f0 0x7f429419b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43710/0 (socket says 172.21.15.102:43710) 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f428be00640 1 -- 172.21.15.102:0/1647792232 learned_addr learned my addr 172.21.15.102:0/1647792232 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f4293400640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42941043b0 0x7f42941a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f4292a00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4294100a20 0x7f429419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f428be00640 1 -- 172.21.15.102:0/1647792232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42941043b0 msgr2=0x7f42941a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f428be00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42941043b0 0x7f42941a21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f428be00640 1 -- 172.21.15.102:0/1647792232 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4294100a20 msgr2=0x7f429419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f428be00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4294100a20 0x7f429419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f428be00640 1 -- 172.21.15.102:0/1647792232 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42941a2860 con 0x7f42941039f0 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.546+0000 7f4293400640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42941043b0 0x7f42941a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.547+0000 7f428be00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42941039f0 0x7f429419b190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f427c00ea60 tx=0x7f427c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.547+0000 7f428b400640 1 -- 172.21.15.102:0/1647792232 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c00cde0 con 0x7f42941039f0 2024-04-26T20:35:08.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.547+0000 7f428b400640 1 -- 172.21.15.102:0/1647792232 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f427c0035f0 con 0x7f42941039f0 2024-04-26T20:35:08.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.547+0000 7f4293e00640 1 -- 172.21.15.102:0/1647792232 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42941a2b50 con 0x7f42941039f0 2024-04-26T20:35:08.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.547+0000 7f428b400640 1 -- 172.21.15.102:0/1647792232 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c010690 con 0x7f42941039f0 2024-04-26T20:35:08.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.547+0000 7f4293e00640 1 -- 172.21.15.102:0/1647792232 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42941a3070 con 0x7f42941039f0 2024-04-26T20:35:08.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f428b400640 1 -- 172.21.15.102:0/1647792232 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f427c0037a0 con 0x7f42941039f0 2024-04-26T20:35:08.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f429c7e4740 1 Processor -- start 2024-04-26T20:35:08.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f429c7e4740 1 -- start start 2024-04-26T20:35:08.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4f2c0 0x561b0ed4f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4fcb0 0x561b0ed547b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed54e20 0x561b0ed572b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f429c7e4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561b0ecb3600 con 0x561b0ed4f2c0 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f429c7e4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561b0ec50100 con 0x561b0ed4fcb0 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f429c7e4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561b0ecaf5c0 con 0x561b0ed54e20 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f4293400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4f2c0 0x561b0ed4f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f4293400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4f2c0 0x561b0ed4f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43726/0 (socket says 172.21.15.102:43726) 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.551+0000 7f4293400640 1 -- 172.21.15.102:0/2469127946 learned_addr learned my addr 172.21.15.102:0/2469127946 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.552+0000 7f4292a00640 1 --2- 172.21.15.102:0/2469127946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4fcb0 0x561b0ed547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.552+0000 7f428be00640 1 --2- 172.21.15.102:0/2469127946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed54e20 0x561b0ed572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.552+0000 7f4293400640 1 -- 172.21.15.102:0/2469127946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed54e20 msgr2=0x561b0ed572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.552+0000 7f4293400640 1 --2- 172.21.15.102:0/2469127946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed54e20 0x561b0ed572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.552+0000 7f4293400640 1 -- 172.21.15.102:0/2469127946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4fcb0 msgr2=0x561b0ed547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.552+0000 7f4293400640 1 --2- 172.21.15.102:0/2469127946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4fcb0 0x561b0ed547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.552+0000 7f4293400640 1 -- 172.21.15.102:0/2469127946 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b0e9c52d0 con 0x561b0ed4f2c0 2024-04-26T20:35:08.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.552+0000 7f4293400640 1 --2- 172.21.15.102:0/2469127946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4f2c0 0x561b0ed4f6e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f428c0117e0 tx=0x7f428c011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186dad2faf53fa96 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.552+0000 7f428b400640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42780a3c40 0x7f42780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.552+0000 7f4283e00640 1 -- 172.21.15.102:0/2469127946 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428c011310 con 0x561b0ed4f2c0 2024-04-26T20:35:08.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f428b400640 1 -- 172.21.15.102:0/1647792232 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f427c0d0c60 con 0x7f42941039f0 2024-04-26T20:35:08.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f4283e00640 1 -- 172.21.15.102:0/2469127946 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f428c003e20 con 0x561b0ed4f2c0 2024-04-26T20:35:08.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f4283e00640 1 -- 172.21.15.102:0/2469127946 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428c0044f0 con 0x561b0ed4f2c0 2024-04-26T20:35:08.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f4292a00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42780a3c40 0x7f42780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f429c7e4740 1 -- 172.21.15.102:0/2469127946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4f2c0 msgr2=0x561b0ed4f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f429c7e4740 1 --2- 172.21.15.102:0/2469127946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4f2c0 0x561b0ed4f6e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f428c0117e0 tx=0x7f428c011ca0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f429c7e4740 1 -- 172.21.15.102:0/2469127946 shutdown_connections 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f429c7e4740 1 --2- 172.21.15.102:0/2469127946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed54e20 0x561b0ed572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f429c7e4740 1 --2- 172.21.15.102:0/2469127946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4fcb0 0x561b0ed547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f429c7e4740 1 --2- 172.21.15.102:0/2469127946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4f2c0 0x561b0ed4f6e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f429c7e4740 1 -- 172.21.15.102:0/2469127946 >> 172.21.15.102:0/2469127946 conn(0x561b0ece3d10 msgr2=0x561b0ed4ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.553+0000 7f4292a00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42780a3c40 0x7f42780a6100 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4284009900 tx=0x7f4284005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.554+0000 7f429c7e4740 1 -- 172.21.15.102:0/2469127946 shutdown_connections 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.554+0000 7f429c7e4740 1 -- 172.21.15.102:0/2469127946 wait complete. 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.554+0000 7f429c7e4740 1 -- start start 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.554+0000 7f429c7e4740 1 Processor -- start 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 0x561b0ed596e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4fcb0 0x561b0ed59c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed54e20 0x561b0ed60c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f429c7e4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561b0ed57dd0 con 0x561b0ed4fcb0 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f429c7e4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561b0ec3af70 con 0x561b0ed4f2c0 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f429c7e4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561b0ed57f50 con 0x561b0ed54e20 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f429c7e4740 10 client.0 _ll_register_callbacks cb 0x561b0eca9480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f428be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4fcb0 0x561b0ed59c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f4293400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed54e20 0x561b0ed60c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f428be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4fcb0 0x561b0ed59c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43732/0 (socket says 172.21.15.102:43732) 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f428be00640 1 -- 172.21.15.102:0/3959982987 learned_addr learned my addr 172.21.15.102:0/3959982987 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f4292a00640 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 0x561b0ed596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f428be00640 1 -- 172.21.15.102:0/3959982987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed54e20 msgr2=0x561b0ed60c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f428be00640 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed54e20 0x561b0ed60c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f428be00640 1 -- 172.21.15.102:0/3959982987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 msgr2=0x561b0ed596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f428be00640 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 0x561b0ed596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f428be00640 1 -- 172.21.15.102:0/3959982987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b0ea8f6b0 con 0x561b0ed4fcb0 2024-04-26T20:35:08.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.555+0000 7f4292a00640 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 0x561b0ed596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.556+0000 7f428be00640 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4fcb0 0x561b0ed59c20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f427c098e00 tx=0x7f427c003cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcac7adf8cbf55f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.556+0000 7f4280c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.556+0000 7f4280c00640 1 -- 172.21.15.102:0/3959982987 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c09c390 con 0x561b0ed4fcb0 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.556+0000 7f4280c00640 1 -- 172.21.15.102:0/3959982987 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f427c099010 con 0x561b0ed4fcb0 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.556+0000 7f429c7e4740 1 -- 172.21.15.102:0/3959982987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f428c003e20 con 0x561b0ed4fcb0 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.556+0000 7f4280c00640 1 -- 172.21.15.102:0/3959982987 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f427c09ab70 con 0x561b0ed4fcb0 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.556+0000 7f429c7e4740 10 client.15264 fetch_fsmap learned FSMap version 27 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.556+0000 7f429c7e4740 1 -- 172.21.15.102:0/3959982987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561b0ed63cb0 con 0x561b0ed4fcb0 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.556+0000 7f4280c00640 1 -- 172.21.15.102:0/3959982987 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f428c003e20 con 0x561b0ed4fcb0 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.556+0000 7f4280c00640 1 -- 172.21.15.102:0/3959982987 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f427c0870b0 con 0x561b0ed4fcb0 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.557+0000 7f429c7e4740 10 client.15264 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.557+0000 7f429c7e4740 10 client.15264 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.557+0000 7f429c7e4740 20 client.15264 populate_metadata read hostname 'smithi102' 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.557+0000 7f429c7e4740 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x561b0ed66320 0x561b0ed67790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.557+0000 7f429c7e4740 4 client.15264 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:35:08.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.557+0000 7f429c7e4740 1 -- 172.21.15.102:0/3959982987 --> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b0ecc1210 con 0x561b0ed66320 2024-04-26T20:35:08.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.557+0000 7f4292a00640 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x561b0ed66320 0x561b0ed67790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.558+0000 7f4292a00640 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x561b0ed66320 0x561b0ed67790 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=110f3719b7984d05 server_cookie=b33baff35a4d7da in_seq=0 out_seq=0 2024-04-26T20:35:08.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.558+0000 7f4280c00640 10 client.15264 ms_handle_connect on v2:172.21.15.112:6836/82964069 2024-04-26T20:35:08.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f4280c00640 1 -- 172.21.15.102:0/3959982987 <== mds.3 v2:172.21.15.112:6836/82964069 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b0ecc1210 con 0x561b0ed66320 2024-04-26T20:35:08.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f4280c00640 10 client.15264 handle_command_reply: tid=0 2024-04-26T20:35:08.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 client.15264 shutdown 2024-04-26T20:35:08.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 -- 172.21.15.102:0/3959982987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4fcb0 msgr2=0x561b0ed59c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4fcb0 0x561b0ed59c20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f427c098e00 tx=0x7f427c003cf0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 -- 172.21.15.102:0/3959982987 shutdown_connections 2024-04-26T20:35:08.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x561b0ed66320 0x561b0ed67790 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed4fcb0 0x561b0ed59c20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed54e20 0x561b0ed60c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 --2- 172.21.15.102:0/3959982987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 0x561b0ed596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 -- 172.21.15.102:0/3959982987 >> 172.21.15.102:0/3959982987 conn(0x561b0ece3d10 msgr2=0x561b0ed49f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f4280c00640 0 client.15264 ms_handle_reset on v2:172.21.15.112:6836/82964069 2024-04-26T20:35:08.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 -- 172.21.15.102:0/3959982987 shutdown_connections 2024-04-26T20:35:08.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 1 -- 172.21.15.102:0/3959982987 wait complete. 2024-04-26T20:35:08.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.559+0000 7f429c7e4740 20 client.15264 trim_cache size 0 max 16384 2024-04-26T20:35:08.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.579+0000 7f429c7e4740 1 Processor -- start 2024-04-26T20:35:08.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.579+0000 7f429c7e4740 1 -- start start 2024-04-26T20:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.579+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 0x561b0ed49de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.579+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4fcb0 0x561b0ed589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 0x561b0ed58ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f429c7e4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561b0ed65ad0 con 0x561b0ed54e20 2024-04-26T20:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f429c7e4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561b0ed661a0 con 0x561b0ed4f2c0 2024-04-26T20:35:08.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f429c7e4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561b0ed66870 con 0x561b0ed4fcb0 2024-04-26T20:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f428be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4fcb0 0x561b0ed589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f4293400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 0x561b0ed58ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f428be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4fcb0 0x561b0ed589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:34984/0 (socket says 172.21.15.102:34984) 2024-04-26T20:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f428be00640 1 -- 172.21.15.102:0/3675893316 learned_addr learned my addr 172.21.15.102:0/3675893316 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f4292a00640 1 --2- 172.21.15.102:0/3675893316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 0x561b0ed49de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f4292a00640 1 -- 172.21.15.102:0/3675893316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4fcb0 msgr2=0x561b0ed589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f4292a00640 1 --2- 172.21.15.102:0/3675893316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4fcb0 0x561b0ed589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f4292a00640 1 -- 172.21.15.102:0/3675893316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 msgr2=0x561b0ed58ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f4292a00640 1 --2- 172.21.15.102:0/3675893316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 0x561b0ed58ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f4292a00640 1 -- 172.21.15.102:0/3675893316 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b0ed50090 con 0x561b0ed4f2c0 2024-04-26T20:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.580+0000 7f428be00640 1 --2- 172.21.15.102:0/3675893316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4fcb0 0x561b0ed589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.581+0000 7f4292a00640 1 --2- 172.21.15.102:0/3675893316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 0x561b0ed49de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4284009230 tx=0x7f4284040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7df555e8f3e528b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.581+0000 7f4283400640 1 -- 172.21.15.102:0/3675893316 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4284057070 con 0x561b0ed4f2c0 2024-04-26T20:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.581+0000 7f4283400640 1 -- 172.21.15.102:0/3675893316 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42840518c0 con 0x561b0ed4f2c0 2024-04-26T20:35:08.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.581+0000 7f4283400640 1 -- 172.21.15.102:0/3675893316 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4284035040 con 0x561b0ed4f2c0 2024-04-26T20:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 -- 172.21.15.102:0/3675893316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 msgr2=0x561b0ed49de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 --2- 172.21.15.102:0/3675893316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 0x561b0ed49de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4284009230 tx=0x7f4284040000 comp rx=0 tx=0).stop 2024-04-26T20:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 -- 172.21.15.102:0/3675893316 shutdown_connections 2024-04-26T20:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 --2- 172.21.15.102:0/3675893316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 0x561b0ed58ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 --2- 172.21.15.102:0/3675893316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4fcb0 0x561b0ed589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 --2- 172.21.15.102:0/3675893316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4f2c0 0x561b0ed49de0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 -- 172.21.15.102:0/3675893316 >> 172.21.15.102:0/3675893316 conn(0x561b0ece3d10 msgr2=0x561b0ed496d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 -- 172.21.15.102:0/3675893316 shutdown_connections 2024-04-26T20:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 -- 172.21.15.102:0/3675893316 wait complete. 2024-04-26T20:35:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 -- start start 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.582+0000 7f429c7e4740 1 Processor -- start 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4f2c0 0x561b0ed5cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4fcb0 0x561b0ed5d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f429c7e4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 0x561b0ed5fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f429c7e4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561b0ed67610 con 0x561b0ed54e20 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f429c7e4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561b0ed66f40 con 0x561b0ed4fcb0 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f429c7e4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561b0ed67ce0 con 0x561b0ed4f2c0 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f429c7e4740 10 client.0 _ll_register_callbacks cb 0x7f427c004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f4293400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 0x561b0ed5fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f4293400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 0x561b0ed5fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43754/0 (socket says 172.21.15.102:43754) 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f4293400640 1 -- 172.21.15.102:0/169845663 learned_addr learned my addr 172.21.15.102:0/169845663 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f4292a00640 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4f2c0 0x561b0ed5cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f428be00640 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4fcb0 0x561b0ed5d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f4293400640 1 -- 172.21.15.102:0/169845663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4f2c0 msgr2=0x561b0ed5cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f4293400640 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4f2c0 0x561b0ed5cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f4293400640 1 -- 172.21.15.102:0/169845663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4fcb0 msgr2=0x561b0ed5d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f4293400640 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4fcb0 0x561b0ed5d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.583+0000 7f4293400640 1 -- 172.21.15.102:0/169845663 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b0ed60630 con 0x561b0ed54e20 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.584+0000 7f4293400640 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 0x561b0ed5fff0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f428c0117b0 tx=0x7f428c003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42302228c365e3b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.584+0000 7f4280c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.584+0000 7f4280c00640 1 -- 172.21.15.102:0/169845663 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f428c00fba0 con 0x561b0ed54e20 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.584+0000 7f4280c00640 1 -- 172.21.15.102:0/169845663 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f428c00fd40 con 0x561b0ed54e20 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.584+0000 7f429c7e4740 1 -- 172.21.15.102:0/169845663 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42840518c0 con 0x561b0ed54e20 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.584+0000 7f4280c00640 1 -- 172.21.15.102:0/169845663 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f428c033050 con 0x561b0ed54e20 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.584+0000 7f429c7e4740 10 client.15270 fetch_fsmap learned FSMap version 27 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.584+0000 7f429c7e4740 1 -- 172.21.15.102:0/169845663 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561b0edccc10 con 0x561b0ed54e20 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.585+0000 7f4280c00640 1 -- 172.21.15.102:0/169845663 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42840518c0 con 0x561b0ed54e20 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.585+0000 7f4280c00640 1 -- 172.21.15.102:0/169845663 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f428c018020 con 0x561b0ed54e20 2024-04-26T20:35:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.585+0000 7f429c7e4740 10 client.15270 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:35:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.585+0000 7f429c7e4740 10 client.15270 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T20:35:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.585+0000 7f429c7e4740 20 client.15270 populate_metadata read hostname 'smithi102' 2024-04-26T20:35:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.585+0000 7f429c7e4740 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x561b0edd1590 0x561b0edce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.585+0000 7f429c7e4740 4 client.15270 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:35:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.585+0000 7f429c7e4740 1 -- 172.21.15.102:0/169845663 --> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] -- command(tid 0: {"prefix": "status"}) -- 0x561b0ecbb080 con 0x561b0edd1590 2024-04-26T20:35:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.585+0000 7f4292a00640 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x561b0edd1590 0x561b0edce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f4292a00640 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x561b0edd1590 0x561b0edce0a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f6ea2ec235a2db22 server_cookie=45988be6adb0b21d in_seq=0 out_seq=0 2024-04-26T20:35:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f4280c00640 10 client.15270 ms_handle_connect on v2:172.21.15.112:6836/82964069 2024-04-26T20:35:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f4280c00640 1 -- 172.21.15.102:0/169845663 <== mds.3 v2:172.21.15.112:6836/82964069 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561b0ecbb080 con 0x561b0edd1590 2024-04-26T20:35:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f4280c00640 10 client.15270 handle_command_reply: tid=0 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f429c7e4740 1 client.15270 shutdown 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f429c7e4740 1 -- 172.21.15.102:0/169845663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 msgr2=0x561b0ed5fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f429c7e4740 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 0x561b0ed5fff0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f428c0117b0 tx=0x7f428c003b80 comp rx=0 tx=0).stop 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f429c7e4740 1 -- 172.21.15.102:0/169845663 shutdown_connections 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f429c7e4740 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.112:6836/82964069,v1:172.21.15.112:6837/82964069] conn(0x561b0edd1590 0x561b0edce0a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f429c7e4740 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561b0ed54e20 0x561b0ed5fff0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f429c7e4740 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561b0ed4fcb0 0x561b0ed5d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f429c7e4740 1 --2- 172.21.15.102:0/169845663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b0ed4f2c0 0x561b0ed5cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f4280c00640 0 client.15270 ms_handle_reset on v2:172.21.15.112:6836/82964069 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.586+0000 7f429c7e4740 1 -- 172.21.15.102:0/169845663 >> 172.21.15.102:0/169845663 conn(0x561b0ece3d10 msgr2=0x561b0ed49f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f429c7e4740 1 -- 172.21.15.102:0/169845663 shutdown_connections 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f429c7e4740 1 -- 172.21.15.102:0/169845663 wait complete. 2024-04-26T20:35:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f429c7e4740 20 client.15270 trim_cache size 0 max 16384 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 3, 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24533, 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 134.92020878599999, 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 27, 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 182.73559536400001 2024-04-26T20:35:08.592 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:35:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f4293e00640 1 -- 172.21.15.102:0/1647792232 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42780a3c40 msgr2=0x7f42780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f4293e00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42780a3c40 0x7f42780a6100 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4284009900 tx=0x7f4284005a20 comp rx=0 tx=0).stop 2024-04-26T20:35:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f4293e00640 1 -- 172.21.15.102:0/1647792232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42941039f0 msgr2=0x7f429419b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f4293e00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42941039f0 0x7f429419b190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f427c00ea60 tx=0x7f427c00ef20 comp rx=0 tx=0).stop 2024-04-26T20:35:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f4293e00640 1 -- 172.21.15.102:0/1647792232 shutdown_connections 2024-04-26T20:35:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f4293e00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f42780a3c40 0x7f42780a6100 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f4293e00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42941039f0 0x7f429419b190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f4293e00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42941043b0 0x7f42941a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f4293e00640 1 --2- 172.21.15.102:0/1647792232 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4294100a20 0x7f429419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.587+0000 7f4293e00640 1 -- 172.21.15.102:0/1647792232 >> 172.21.15.102:0/1647792232 conn(0x7f42940fc350 msgr2=0x7f42941017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.588+0000 7f4293e00640 1 -- 172.21.15.102:0/1647792232 shutdown_connections 2024-04-26T20:35:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.588+0000 7f4293e00640 1 -- 172.21.15.102:0/1647792232 wait complete. 2024-04-26T20:35:08.605 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-26T20:35:08.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.798+0000 7f0ea0800640 1 -- 172.21.15.102:0/2985102900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e9810a1a0 msgr2=0x7f0e9810a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.798+0000 7f0ea0800640 1 --2- 172.21.15.102:0/2985102900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e9810a1a0 0x7f0e9810a620 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009900 tx=0x7f0e8802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.798+0000 7f0ea0800640 1 -- 172.21.15.102:0/2985102900 shutdown_connections 2024-04-26T20:35:08.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.798+0000 7f0ea0800640 1 --2- 172.21.15.102:0/2985102900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e9810a1a0 0x7f0e9810a620 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.798+0000 7f0ea0800640 1 --2- 172.21.15.102:0/2985102900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e9810ab60 0x7f0e981113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.798+0000 7f0ea0800640 1 --2- 172.21.15.102:0/2985102900 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e98108fb0 0x7f0e981093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.798+0000 7f0ea0800640 1 -- 172.21.15.102:0/2985102900 >> 172.21.15.102:0/2985102900 conn(0x7f0e98077f40 msgr2=0x7f0e9807a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.798+0000 7f0ea0800640 1 -- 172.21.15.102:0/2985102900 shutdown_connections 2024-04-26T20:35:08.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.798+0000 7f0ea0800640 1 -- 172.21.15.102:0/2985102900 wait complete. 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.799+0000 7f0ea0800640 1 Processor -- start 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.799+0000 7f0ea0800640 1 -- start start 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.799+0000 7f0ea0800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e98108fb0 0x7f0e9819f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.799+0000 7f0ea0800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e9810a1a0 0x7f0e9819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.799+0000 7f0ea0800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e9810ab60 0x7f0e981a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.799+0000 7f0ea0800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0e98114af0 con 0x7f0e9810ab60 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.799+0000 7f0ea0800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0e98114970 con 0x7f0e98108fb0 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.799+0000 7f0ea0800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0e98114c70 con 0x7f0e9810a1a0 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e98108fb0 0x7f0e9819f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e9810ab60 0x7f0e981a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e9810ab60 0x7f0e981a6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43772/0 (socket says 172.21.15.102:43772) 2024-04-26T20:35:08.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e98108fb0 0x7f0e9819f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:60462/0 (socket says 172.21.15.102:60462) 2024-04-26T20:35:08.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9e800640 1 -- 172.21.15.102:0/2112640370 learned_addr learned my addr 172.21.15.102:0/2112640370 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9d400640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e9810a1a0 0x7f0e9819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9e800640 1 -- 172.21.15.102:0/2112640370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e9810a1a0 msgr2=0x7f0e9819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9e800640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e9810a1a0 0x7f0e9819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9e800640 1 -- 172.21.15.102:0/2112640370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e98108fb0 msgr2=0x7f0e9819f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9e800640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e98108fb0 0x7f0e9819f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.800+0000 7f0e9e800640 1 -- 172.21.15.102:0/2112640370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e981a6e00 con 0x7f0e9810ab60 2024-04-26T20:35:08.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.801+0000 7f0e9e800640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e9810ab60 0x7f0e981a6790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400efd0 tx=0x7f0e9400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.801+0000 7f0e92a00640 1 -- 172.21.15.102:0/2112640370 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94003b70 con 0x7f0e9810ab60 2024-04-26T20:35:08.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.801+0000 7f0e92a00640 1 -- 172.21.15.102:0/2112640370 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e94003d10 con 0x7f0e9810ab60 2024-04-26T20:35:08.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.801+0000 7f0e92a00640 1 -- 172.21.15.102:0/2112640370 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e940108a0 con 0x7f0e9810ab60 2024-04-26T20:35:08.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.801+0000 7f0ea0800640 1 -- 172.21.15.102:0/2112640370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e981a70f0 con 0x7f0e9810ab60 2024-04-26T20:35:08.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.801+0000 7f0ea0800640 1 -- 172.21.15.102:0/2112640370 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e981a75c0 con 0x7f0e9810ab60 2024-04-26T20:35:08.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.804+0000 7f0ea3601740 1 Processor -- start 2024-04-26T20:35:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.804+0000 7f0e92a00640 1 -- 172.21.15.102:0/2112640370 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f0e94004020 con 0x7f0e9810ab60 2024-04-26T20:35:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0ea3601740 1 -- start start 2024-04-26T20:35:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 0x56084a6ec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 0x56084a6f15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0ea3601740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56084a650600 con 0x56084a6f1c50 2024-04-26T20:35:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0ea3601740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56084a5ed100 con 0x56084a6ec0f0 2024-04-26T20:35:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0ea3601740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56084a64c5c0 con 0x56084a6ecae0 2024-04-26T20:35:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0e9de00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 0x56084a6f15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0e9de00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 0x56084a6f15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:35024/0 (socket says 172.21.15.102:35024) 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0e9de00640 1 -- 172.21.15.102:0/1750531460 learned_addr learned my addr 172.21.15.102:0/1750531460 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0e9d400640 1 --2- 172.21.15.102:0/1750531460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.805+0000 7f0e9d400640 1 --2- 172.21.15.102:0/1750531460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 0x56084a6ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.806+0000 7f0e9de00640 1 -- 172.21.15.102:0/1750531460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 msgr2=0x56084a6ec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.806+0000 7f0e9de00640 1 --2- 172.21.15.102:0/1750531460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 0x56084a6ec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.806+0000 7f0e9de00640 1 -- 172.21.15.102:0/1750531460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 msgr2=0x56084a6f40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.806+0000 7f0e9de00640 1 --2- 172.21.15.102:0/1750531460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6f40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.806+0000 7f0e9de00640 1 -- 172.21.15.102:0/1750531460 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56084a3622d0 con 0x56084a6ecae0 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.806+0000 7f0e9de00640 1 --2- 172.21.15.102:0/1750531460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 0x56084a6f15e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009ca0 tx=0x7f0e8c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14e33a2d507470bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.806+0000 7f0e87000640 1 -- 172.21.15.102:0/1750531460 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c004220 con 0x56084a6ecae0 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.806+0000 7f0e87000640 1 -- 172.21.15.102:0/1750531460 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e8c0043c0 con 0x56084a6ecae0 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.806+0000 7f0e92a00640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0e800a3cc0 0x7f0e800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.806+0000 7f0e87000640 1 -- 172.21.15.102:0/1750531460 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c019830 con 0x56084a6ecae0 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0e92a00640 1 -- 172.21.15.102:0/2112640370 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f0e940d0d20 con 0x7f0e9810ab60 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0e9d400640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0e800a3cc0 0x7f0e800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0ea3601740 1 -- 172.21.15.102:0/1750531460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 msgr2=0x56084a6f15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0ea3601740 1 --2- 172.21.15.102:0/1750531460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 0x56084a6f15e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009ca0 tx=0x7f0e8c011810 comp rx=0 tx=0).stop 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0ea3601740 1 -- 172.21.15.102:0/1750531460 shutdown_connections 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0ea3601740 1 --2- 172.21.15.102:0/1750531460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0ea3601740 1 --2- 172.21.15.102:0/1750531460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 0x56084a6f15e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0ea3601740 1 --2- 172.21.15.102:0/1750531460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 0x56084a6ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0ea3601740 1 -- 172.21.15.102:0/1750531460 >> 172.21.15.102:0/1750531460 conn(0x56084a680d10 msgr2=0x56084a6e7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0ea3601740 1 -- 172.21.15.102:0/1750531460 shutdown_connections 2024-04-26T20:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0ea3601740 1 -- 172.21.15.102:0/1750531460 wait complete. 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.807+0000 7f0e9d400640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0e800a3cc0 0x7f0e800a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0e88000c00 tx=0x7f0e8803f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.808+0000 7f0ea3601740 1 -- start start 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.808+0000 7f0ea3601740 1 Processor -- start 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.808+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 0x56084a6f6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.808+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 0x56084a6f6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.808+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6fdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.808+0000 7f0ea3601740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56084a6f4c00 con 0x56084a6f1c50 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.808+0000 7f0ea3601740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56084a5d7f70 con 0x56084a6ec0f0 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.808+0000 7f0ea3601740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56084a6f4d80 con 0x56084a6ecae0 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.808+0000 7f0ea3601740 10 client.0 _ll_register_callbacks cb 0x56084a646480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6fdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 0x56084a6f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9d400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 0x56084a6f6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 0x56084a6f6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:60482/0 (socket says 172.21.15.102:60482) 2024-04-26T20:35:08.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9de00640 1 -- 172.21.15.102:0/3342783077 learned_addr learned my addr 172.21.15.102:0/3342783077 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9e800640 1 -- 172.21.15.102:0/3342783077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 msgr2=0x56084a6f6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9e800640 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 0x56084a6f6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9e800640 1 -- 172.21.15.102:0/3342783077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 msgr2=0x56084a6f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9e800640 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 0x56084a6f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9e800640 1 -- 172.21.15.102:0/3342783077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56084a42c6b0 con 0x56084a6f1c50 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9de00640 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 0x56084a6f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e9e800640 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6fdaf0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0e940ce2c0 tx=0x7f0e94009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6befbfc2626facaf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.809+0000 7f0e7b400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0e7b400640 1 -- 172.21.15.102:0/3342783077 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9409ca90 con 0x56084a6f1c50 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0e7b400640 1 -- 172.21.15.102:0/3342783077 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f0e940ce510 con 0x56084a6f1c50 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0ea3601740 1 -- 172.21.15.102:0/3342783077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e8c0043c0 con 0x56084a6f1c50 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0e7b400640 1 -- 172.21.15.102:0/3342783077 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0e940a0e40 con 0x56084a6f1c50 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0ea3601740 10 client.15282 fetch_fsmap learned FSMap version 27 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0ea3601740 1 -- 172.21.15.102:0/3342783077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56084a700ba0 con 0x56084a6f1c50 2024-04-26T20:35:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0e7b400640 1 -- 172.21.15.102:0/3342783077 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e8c0043c0 con 0x56084a6f1c50 2024-04-26T20:35:08.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0e7b400640 1 -- 172.21.15.102:0/3342783077 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0e9401f020 con 0x56084a6f1c50 2024-04-26T20:35:08.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0ea3601740 10 client.15282 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:35:08.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0ea3601740 10 client.15282 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T20:35:08.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0ea3601740 20 client.15282 populate_metadata read hostname 'smithi102' 2024-04-26T20:35:08.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x56084a7031e0 0x56084a704650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0ea3601740 4 client.15282 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:35:08.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.810+0000 7f0ea3601740 1 -- 172.21.15.102:0/3342783077 --> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56084a65e210 con 0x56084a7031e0 2024-04-26T20:35:08.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.811+0000 7f0e9de00640 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x56084a7031e0 0x56084a704650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.811+0000 7f0e9de00640 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x56084a7031e0 0x56084a704650 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fab772fcd8e28771 server_cookie=d3dfacd8b41e6b8c in_seq=0 out_seq=0 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.811+0000 7f0e7b400640 10 client.15282 ms_handle_connect on v2:172.21.15.130:6838/1700008514 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0e7b400640 1 -- 172.21.15.102:0/3342783077 <== mds.2 v2:172.21.15.130:6838/1700008514 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56084a65e210 con 0x56084a7031e0 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0e7b400640 10 client.15282 handle_command_reply: tid=0 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0ea3601740 1 client.15282 shutdown 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0ea3601740 1 -- 172.21.15.102:0/3342783077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 msgr2=0x56084a6fdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6fdaf0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0e940ce2c0 tx=0x7f0e94009820 comp rx=0 tx=0).stop 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0ea3601740 1 -- 172.21.15.102:0/3342783077 shutdown_connections 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x56084a7031e0 0x56084a704650 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6fdaf0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ecae0 0x56084a6f6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0e7b400640 0 client.15282 ms_handle_reset on v2:172.21.15.130:6838/1700008514 2024-04-26T20:35:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3342783077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ec0f0 0x56084a6f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.812+0000 7f0ea3601740 1 -- 172.21.15.102:0/3342783077 >> 172.21.15.102:0/3342783077 conn(0x56084a680d10 msgr2=0x56084a6e6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.813+0000 7f0ea3601740 1 -- 172.21.15.102:0/3342783077 shutdown_connections 2024-04-26T20:35:08.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.813+0000 7f0ea3601740 1 -- 172.21.15.102:0/3342783077 wait complete. 2024-04-26T20:35:08.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.813+0000 7f0ea3601740 20 client.15282 trim_cache size 0 max 16384 2024-04-26T20:35:08.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.832+0000 7f0ea3601740 1 Processor -- start 2024-04-26T20:35:08.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.832+0000 7f0ea3601740 1 -- start start 2024-04-26T20:35:08.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ec0f0 0x56084a6f6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 0x56084a6fdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6ea830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0ea3601740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56084a702990 con 0x56084a6f1c50 2024-04-26T20:35:08.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0ea3601740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56084a703060 con 0x56084a6ecae0 2024-04-26T20:35:08.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0ea3601740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56084a703730 con 0x56084a6ec0f0 2024-04-26T20:35:08.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6ea830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6ea830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43798/0 (socket says 172.21.15.102:43798) 2024-04-26T20:35:08.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9e800640 1 -- 172.21.15.102:0/1268324751 learned_addr learned my addr 172.21.15.102:0/1268324751 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9de00640 1 --2- 172.21.15.102:0/1268324751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ec0f0 0x56084a6f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9d400640 1 --2- 172.21.15.102:0/1268324751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 0x56084a6fdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9e800640 1 -- 172.21.15.102:0/1268324751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ec0f0 msgr2=0x56084a6f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9e800640 1 --2- 172.21.15.102:0/1268324751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ec0f0 0x56084a6f6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9e800640 1 -- 172.21.15.102:0/1268324751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 msgr2=0x56084a6fdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9e800640 1 --2- 172.21.15.102:0/1268324751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 0x56084a6fdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9e800640 1 -- 172.21.15.102:0/1268324751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56084a6e38a0 con 0x56084a6f1c50 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.833+0000 7f0e9d400640 1 --2- 172.21.15.102:0/1268324751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 0x56084a6fdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.834+0000 7f0e9e800640 1 --2- 172.21.15.102:0/1268324751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6ea830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0e94099460 tx=0x7f0e9409afa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81d6aa26615583b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.834+0000 7f0e86600640 1 -- 172.21.15.102:0/1268324751 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9409ca90 con 0x56084a6f1c50 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.834+0000 7f0e86600640 1 -- 172.21.15.102:0/1268324751 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e940a5700 con 0x56084a6f1c50 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.834+0000 7f0e86600640 1 -- 172.21.15.102:0/1268324751 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9409c120 con 0x56084a6f1c50 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 -- 172.21.15.102:0/1268324751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 msgr2=0x56084a6ea830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 --2- 172.21.15.102:0/1268324751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6ea830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0e94099460 tx=0x7f0e9409afa0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 -- 172.21.15.102:0/1268324751 shutdown_connections 2024-04-26T20:35:08.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 --2- 172.21.15.102:0/1268324751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6f1c50 0x56084a6ea830 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 --2- 172.21.15.102:0/1268324751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 0x56084a6fdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 --2- 172.21.15.102:0/1268324751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6ec0f0 0x56084a6f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 -- 172.21.15.102:0/1268324751 >> 172.21.15.102:0/1268324751 conn(0x56084a680d10 msgr2=0x56084a6f6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 -- 172.21.15.102:0/1268324751 shutdown_connections 2024-04-26T20:35:08.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 -- 172.21.15.102:0/1268324751 wait complete. 2024-04-26T20:35:08.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 -- start start 2024-04-26T20:35:08.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.835+0000 7f0ea3601740 1 Processor -- start 2024-04-26T20:35:08.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6ec0f0 0x56084a6f9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 0x56084a6f99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0ea3601740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6f1c50 0x56084a6fc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0ea3601740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56084a7044d0 con 0x56084a6ec0f0 2024-04-26T20:35:08.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0ea3601740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56084a703e00 con 0x56084a6ecae0 2024-04-26T20:35:08.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0ea3601740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56084a704ba0 con 0x56084a6f1c50 2024-04-26T20:35:08.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0ea3601740 10 client.0 _ll_register_callbacks cb 0x7f0e94009b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:35:08.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6ec0f0 0x56084a6f9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6ec0f0 0x56084a6f9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43800/0 (socket says 172.21.15.102:43800) 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9de00640 1 -- 172.21.15.102:0/3699528351 learned_addr learned my addr 172.21.15.102:0/3699528351 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9e800640 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6f1c50 0x56084a6fc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9d400640 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 0x56084a6f99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9de00640 1 -- 172.21.15.102:0/3699528351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6f1c50 msgr2=0x56084a6fc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9de00640 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6f1c50 0x56084a6fc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9de00640 1 -- 172.21.15.102:0/3699528351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 msgr2=0x56084a6f99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9de00640 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 0x56084a6f99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9de00640 1 -- 172.21.15.102:0/3699528351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56084a6fcf80 con 0x56084a6ec0f0 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.836+0000 7f0e9e800640 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6f1c50 0x56084a6fc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.837+0000 7f0e9de00640 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6ec0f0 0x56084a6f9480 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009bc0 tx=0x7f0e8c009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78cfa541ad984f3f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:08.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.837+0000 7f0e7b400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.837+0000 7f0e7b400640 1 -- 172.21.15.102:0/3699528351 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0055a0 con 0x56084a6ec0f0 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.837+0000 7f0e7b400640 1 -- 172.21.15.102:0/3699528351 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f0e8c01f040 con 0x56084a6ec0f0 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.837+0000 7f0ea3601740 1 -- 172.21.15.102:0/3699528351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e940a5700 con 0x56084a6ec0f0 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.837+0000 7f0e7b400640 1 -- 172.21.15.102:0/3699528351 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0e8c03b050 con 0x56084a6ec0f0 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.837+0000 7f0ea3601740 10 client.15288 fetch_fsmap learned FSMap version 27 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.837+0000 7f0ea3601740 1 -- 172.21.15.102:0/3699528351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56084a769410 con 0x56084a6ec0f0 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.838+0000 7f0e7b400640 1 -- 172.21.15.102:0/3699528351 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e940a5700 con 0x56084a6ec0f0 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.838+0000 7f0e7b400640 1 -- 172.21.15.102:0/3699528351 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0e8c02c2f0 con 0x56084a6ec0f0 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.838+0000 7f0ea3601740 10 client.15288 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.838+0000 7f0ea3601740 10 client.15288 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T20:35:08.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.838+0000 7f0ea3601740 20 client.15288 populate_metadata read hostname 'smithi102' 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.838+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x56084a76deb0 0x56084a76cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.838+0000 7f0ea3601740 4 client.15288 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.838+0000 7f0ea3601740 1 -- 172.21.15.102:0/3699528351 --> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] -- command(tid 0: {"prefix": "status"}) -- 0x56084a658080 con 0x56084a76deb0 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.838+0000 7f0e9d400640 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x56084a76deb0 0x56084a76cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0e9d400640 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x56084a76deb0 0x56084a76cbc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=30384713bca43ae5 server_cookie=938879b5fbbdb37c in_seq=0 out_seq=0 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0e7b400640 10 client.15288 ms_handle_connect on v2:172.21.15.130:6838/1700008514 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0e7b400640 1 -- 172.21.15.102:0/3699528351 <== mds.2 v2:172.21.15.130:6838/1700008514 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x56084a658080 con 0x56084a76deb0 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0e7b400640 10 client.15288 handle_command_reply: tid=0 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 client.15288 shutdown 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 -- 172.21.15.102:0/3699528351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6ec0f0 msgr2=0x56084a6f9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6ec0f0 0x56084a6f9480 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009bc0 tx=0x7f0e8c009f10 comp rx=0 tx=0).stop 2024-04-26T20:35:08.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 -- 172.21.15.102:0/3699528351 shutdown_connections 2024-04-26T20:35:08.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.130:6838/1700008514,v1:172.21.15.130:6839/1700008514] conn(0x56084a76deb0 0x56084a76cbc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56084a6f1c50 0x56084a6fc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56084a6ecae0 0x56084a6f99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 --2- 172.21.15.102:0/3699528351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56084a6ec0f0 0x56084a6f9480 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 -- 172.21.15.102:0/3699528351 >> 172.21.15.102:0/3699528351 conn(0x56084a680d10 msgr2=0x56084a6f5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0e7b400640 0 client.15288 ms_handle_reset on v2:172.21.15.130:6838/1700008514 2024-04-26T20:35:08.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 -- 172.21.15.102:0/3699528351 shutdown_connections 2024-04-26T20:35:08.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 1 -- 172.21.15.102:0/3699528351 wait complete. 2024-04-26T20:35:08.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.839+0000 7f0ea3601740 20 client.15288 trim_cache size 0 max 16384 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 2, 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24547, 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 136.184913932, 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 27, 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-04-26T20:35:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:35:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 180.130796046 2024-04-26T20:35:08.848 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:35:08.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 -- 172.21.15.102:0/2112640370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0e800a3cc0 msgr2=0x7f0e800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0e800a3cc0 0x7f0e800a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0e88000c00 tx=0x7f0e8803f040 comp rx=0 tx=0).stop 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 -- 172.21.15.102:0/2112640370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e9810ab60 msgr2=0x7f0e981a6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e9810ab60 0x7f0e981a6790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400efd0 tx=0x7f0e9400c6a0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 -- 172.21.15.102:0/2112640370 shutdown_connections 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f0e800a3cc0 0x7f0e800a6180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e9810ab60 0x7f0e981a6790 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e9810a1a0 0x7f0e9819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 --2- 172.21.15.102:0/2112640370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e98108fb0 0x7f0e9819f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 -- 172.21.15.102:0/2112640370 >> 172.21.15.102:0/2112640370 conn(0x7f0e98077f40 msgr2=0x7f0e9810d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 -- 172.21.15.102:0/2112640370 shutdown_connections 2024-04-26T20:35:08.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:08.840+0000 7f0ea0800640 1 -- 172.21.15.102:0/2112640370 wait complete. 2024-04-26T20:35:08.856 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:08 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2982538165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:35:08.856 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:08 smithi102 ceph-mon[28094]: pgmap v403: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:35:08.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-26T20:35:09.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:08 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2982538165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:35:09.087 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:08 smithi112 ceph-mon[32769]: pgmap v403: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:35:09.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.049+0000 7fc4ea400640 1 -- 172.21.15.102:0/964393871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4e40772f0 msgr2=0x7fc4e4077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.049+0000 7fc4ea400640 1 --2- 172.21.15.102:0/964393871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4e40772f0 0x7fc4e4077710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009900 tx=0x7fc4d802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.049+0000 7fc4ea400640 1 -- 172.21.15.102:0/964393871 shutdown_connections 2024-04-26T20:35:09.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.049+0000 7fc4ea400640 1 --2- 172.21.15.102:0/964393871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4e4078ed0 0x7fc4e4079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.049+0000 7fc4ea400640 1 --2- 172.21.15.102:0/964393871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4e4078530 0x7fc4e4078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.049+0000 7fc4ea400640 1 --2- 172.21.15.102:0/964393871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4e40772f0 0x7fc4e4077710 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.049+0000 7fc4ea400640 1 -- 172.21.15.102:0/964393871 >> 172.21.15.102:0/964393871 conn(0x7fc4e40fff10 msgr2=0x7fc4e4102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.049+0000 7fc4ea400640 1 -- 172.21.15.102:0/964393871 shutdown_connections 2024-04-26T20:35:09.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.049+0000 7fc4ea400640 1 -- 172.21.15.102:0/964393871 wait complete. 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.050+0000 7fc4ea400640 1 Processor -- start 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.050+0000 7fc4ea400640 1 -- start start 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4ea400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4e40772f0 0x7fc4e419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4ea400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4e4078530 0x7fc4e419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4ea400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4e4078ed0 0x7fc4e41a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4ea400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4114900 con 0x7fc4e4078530 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4ea400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4114780 con 0x7fc4e4078ed0 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4ea400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4114a80 con 0x7fc4e40772f0 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4e9000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4e40772f0 0x7fc4e419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4e3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4e4078530 0x7fc4e419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4e9000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4e40772f0 0x7fc4e419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:35076/0 (socket says 172.21.15.102:35076) 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4e9000640 1 -- 172.21.15.102:0/1711546891 learned_addr learned my addr 172.21.15.102:0/1711546891 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:09.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4e9a00640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4e4078ed0 0x7fc4e41a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4e9000640 1 -- 172.21.15.102:0/1711546891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4e4078ed0 msgr2=0x7fc4e41a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4e9000640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4e4078ed0 0x7fc4e41a6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4e9000640 1 -- 172.21.15.102:0/1711546891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4e4078530 msgr2=0x7fc4e419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4e9000640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4e4078530 0x7fc4e419f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.051+0000 7fc4e9000640 1 -- 172.21.15.102:0/1711546891 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e41a6be0 con 0x7fc4e40772f0 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.052+0000 7fc4e9000640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4e40772f0 0x7fc4e419efc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8005ab0 tx=0x7fc4d8003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.052+0000 7fc4e1600640 1 -- 172.21.15.102:0/1711546891 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d803f070 con 0x7fc4e40772f0 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.052+0000 7fc4e1600640 1 -- 172.21.15.102:0/1711546891 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4d8031c50 con 0x7fc4e40772f0 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.052+0000 7fc4ea400640 1 -- 172.21.15.102:0/1711546891 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4e41a6e70 con 0x7fc4e40772f0 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.052+0000 7fc4e1600640 1 -- 172.21.15.102:0/1711546891 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8035030 con 0x7fc4e40772f0 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.052+0000 7fc4ea400640 1 -- 172.21.15.102:0/1711546891 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4e41a7270 con 0x7fc4e40772f0 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.055+0000 7fc4ee84e740 1 Processor -- start 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4e1600640 1 -- 172.21.15.102:0/1711546891 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc4d8042070 con 0x7fc4e40772f0 2024-04-26T20:35:09.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4ee84e740 1 -- start start 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403746e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640374cb0 0x5616403797b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 0x56164037c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5616402d8600 con 0x5616403742c0 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561640275100 con 0x561640374cb0 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5616402d45c0 con 0x561640379e20 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4e9a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403746e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4e3e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 0x56164037c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4e9000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640374cb0 0x5616403797b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4e3e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 0x56164037c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:35080/0 (socket says 172.21.15.102:35080) 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4e3e00640 1 -- 172.21.15.102:0/714915123 learned_addr learned my addr 172.21.15.102:0/714915123 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.056+0000 7fc4e9a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403746e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43824/0 (socket says 172.21.15.102:43824) 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.057+0000 7fc4e3e00640 1 -- 172.21.15.102:0/714915123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640374cb0 msgr2=0x5616403797b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.057+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/714915123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640374cb0 0x5616403797b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.057+0000 7fc4e3e00640 1 -- 172.21.15.102:0/714915123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 msgr2=0x5616403746e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.057+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/714915123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403746e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.057+0000 7fc4e3e00640 1 -- 172.21.15.102:0/714915123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56163ffea2d0 con 0x561640379e20 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.057+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/714915123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 0x56164037c2b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009ca0 tx=0x7fc4d0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44e397fb6090acd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.057+0000 7fc4d6000640 1 -- 172.21.15.102:0/714915123 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d001e070 con 0x561640379e20 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.058+0000 7fc4d6000640 1 -- 172.21.15.102:0/714915123 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4d0019ae0 con 0x561640379e20 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.058+0000 7fc4d6000640 1 -- 172.21.15.102:0/714915123 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d000e950 con 0x561640379e20 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.058+0000 7fc4e1600640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4cc0a3c40 0x7fc4cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.058+0000 7fc4e9a00640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4cc0a3c40 0x7fc4cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.058+0000 7fc4e1600640 1 -- 172.21.15.102:0/1711546891 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc4d80f1140 con 0x7fc4e40772f0 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.058+0000 7fc4ee84e740 1 -- 172.21.15.102:0/714915123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 msgr2=0x56164037c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.058+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/714915123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 0x56164037c2b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009ca0 tx=0x7fc4d0011810 comp rx=0 tx=0).stop 2024-04-26T20:35:09.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.058+0000 7fc4e9a00640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4cc0a3c40 0x7fc4cc0a6100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x561640375510 tx=0x7fc4dc0096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.059+0000 7fc4ee84e740 1 -- 172.21.15.102:0/714915123 shutdown_connections 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.059+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/714915123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 0x56164037c2b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.059+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/714915123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640374cb0 0x5616403797b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.059+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/714915123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.059+0000 7fc4ee84e740 1 -- 172.21.15.102:0/714915123 >> 172.21.15.102:0/714915123 conn(0x561640308d10 msgr2=0x56164036fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.059+0000 7fc4ee84e740 1 -- 172.21.15.102:0/714915123 shutdown_connections 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.059+0000 7fc4ee84e740 1 -- 172.21.15.102:0/714915123 wait complete. 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.059+0000 7fc4ee84e740 1 -- start start 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.059+0000 7fc4ee84e740 1 Processor -- start 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x56164037e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 0x56164037ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 0x561640385c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56164037cdd0 con 0x5616403742c0 2024-04-26T20:35:09.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56164025ff70 con 0x561640379e20 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56164037cf50 con 0x561640374cb0 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4ee84e740 10 client.0 _ll_register_callbacks cb 0x5616402ce480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4e9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x56164037e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4e9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x56164037e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43832/0 (socket says 172.21.15.102:43832) 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4e9000640 1 -- 172.21.15.102:0/2792724294 learned_addr learned my addr 172.21.15.102:0/2792724294 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 0x56164037ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.060+0000 7fc4e9a00640 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 0x561640385c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4e9000640 1 -- 172.21.15.102:0/2792724294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 msgr2=0x56164037ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4e9000640 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 0x56164037ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4e9000640 1 -- 172.21.15.102:0/2792724294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 msgr2=0x561640385c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4e9000640 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 0x561640385c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4e9000640 1 -- 172.21.15.102:0/2792724294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616400b46b0 con 0x5616403742c0 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 0x56164037ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:09.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4e9a00640 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 0x561640385c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4e9000640 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x56164037e6e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80ee710 tx=0x7fc4d8034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2f215959b1c417 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4caa00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4caa00640 1 -- 172.21.15.102:0/2792724294 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8049070 con 0x5616403742c0 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4caa00640 1 -- 172.21.15.102:0/2792724294 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc4d80ee930 con 0x5616403742c0 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4ee84e740 1 -- 172.21.15.102:0/2792724294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4d0019ae0 con 0x5616403742c0 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4caa00640 1 -- 172.21.15.102:0/2792724294 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc4d80bcaa0 con 0x5616403742c0 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4ee84e740 10 client.15300 fetch_fsmap learned FSMap version 27 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.061+0000 7fc4ee84e740 1 -- 172.21.15.102:0/2792724294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561640388c40 con 0x5616403742c0 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.062+0000 7fc4caa00640 1 -- 172.21.15.102:0/2792724294 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4d0019ae0 con 0x5616403742c0 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.062+0000 7fc4caa00640 1 -- 172.21.15.102:0/2792724294 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc4d80bd1c0 con 0x5616403742c0 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.062+0000 7fc4ee84e740 10 client.15300 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.062+0000 7fc4ee84e740 10 client.15300 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.062+0000 7fc4ee84e740 20 client.15300 populate_metadata read hostname 'smithi102' 2024-04-26T20:35:09.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.062+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x56164038b2a0 0x56164038c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.062+0000 7fc4ee84e740 4 client.15300 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.062+0000 7fc4ee84e740 1 -- 172.21.15.102:0/2792724294 --> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616402e6210 con 0x56164038b2a0 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.062+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x56164038b2a0 0x56164038c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.063+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x56164038b2a0 0x56164038c710 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac3fa94c5e0bd37d server_cookie=31f70da0dd640a17 in_seq=0 out_seq=0 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.063+0000 7fc4caa00640 10 client.15300 ms_handle_connect on v2:172.21.15.102:6838/3473487681 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4caa00640 1 -- 172.21.15.102:0/2792724294 <== mds.1 v2:172.21.15.102:6838/3473487681 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616402e6210 con 0x56164038b2a0 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4caa00640 10 client.15300 handle_command_reply: tid=0 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 client.15300 shutdown 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 -- 172.21.15.102:0/2792724294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 msgr2=0x56164037e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x56164037e6e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80ee710 tx=0x7fc4d8034000 comp rx=0 tx=0).stop 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 -- 172.21.15.102:0/2792724294 shutdown_connections 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x56164038b2a0 0x56164038c710 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 0x561640385c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 0x56164037ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4caa00640 0 client.15300 ms_handle_reset on v2:172.21.15.102:6838/3473487681 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/2792724294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x56164037e6e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 -- 172.21.15.102:0/2792724294 >> 172.21.15.102:0/2792724294 conn(0x561640308d10 msgr2=0x56164036ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 -- 172.21.15.102:0/2792724294 shutdown_connections 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 1 -- 172.21.15.102:0/2792724294 wait complete. 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.064+0000 7fc4ee84e740 20 client.15300 trim_cache size 0 max 16384 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.084+0000 7fc4ee84e740 1 Processor -- start 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.084+0000 7fc4ee84e740 1 -- start start 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.084+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403796e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.084+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 0x561640385c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 0x561640376070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56164038aa50 con 0x5616403742c0 2024-04-26T20:35:09.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56164038b120 con 0x561640379e20 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56164038b7f0 con 0x561640374cb0 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4e9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403796e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4e9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403796e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43846/0 (socket says 172.21.15.102:43846) 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4e9000640 1 -- 172.21.15.102:0/3084075928 learned_addr learned my addr 172.21.15.102:0/3084075928 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/3084075928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 0x561640385c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4e9a00640 1 --2- 172.21.15.102:0/3084075928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 0x561640376070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4e9000640 1 -- 172.21.15.102:0/3084075928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 msgr2=0x561640385c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4e9000640 1 --2- 172.21.15.102:0/3084075928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 0x561640385c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4e9000640 1 -- 172.21.15.102:0/3084075928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 msgr2=0x561640376070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4e9000640 1 --2- 172.21.15.102:0/3084075928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 0x561640376070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.085+0000 7fc4e9000640 1 -- 172.21.15.102:0/3084075928 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616403709f0 con 0x5616403742c0 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.086+0000 7fc4e9000640 1 --2- 172.21.15.102:0/3084075928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403796e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80b93c0 tx=0x7fc4d80bb470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d302debd7906ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.086+0000 7fc4d5600640 1 -- 172.21.15.102:0/3084075928 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8049070 con 0x5616403742c0 2024-04-26T20:35:09.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.086+0000 7fc4d5600640 1 -- 172.21.15.102:0/3084075928 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4d80c1260 con 0x5616403742c0 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.086+0000 7fc4d5600640 1 -- 172.21.15.102:0/3084075928 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d80bc8c0 con 0x5616403742c0 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.086+0000 7fc4ee84e740 1 -- 172.21.15.102:0/3084075928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 msgr2=0x5616403796e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.086+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/3084075928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403796e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80b93c0 tx=0x7fc4d80bb470 comp rx=0 tx=0).stop 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.087+0000 7fc4ee84e740 1 -- 172.21.15.102:0/3084075928 shutdown_connections 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.087+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/3084075928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561640379e20 0x561640376070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.087+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/3084075928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640374cb0 0x561640385c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.087+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/3084075928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5616403742c0 0x5616403796e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.087+0000 7fc4ee84e740 1 -- 172.21.15.102:0/3084075928 >> 172.21.15.102:0/3084075928 conn(0x561640308d10 msgr2=0x561640389f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.087+0000 7fc4ee84e740 1 -- 172.21.15.102:0/3084075928 shutdown_connections 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.087+0000 7fc4ee84e740 1 -- 172.21.15.102:0/3084075928 wait complete. 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.087+0000 7fc4ee84e740 1 -- start start 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.087+0000 7fc4ee84e740 1 Processor -- start 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5616403742c0 0x5616403814d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561640374cb0 0x561640381a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4ee84e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 0x561640384a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56164038c590 con 0x561640374cb0 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56164038bec0 con 0x5616403742c0 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4ee84e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56164038cc60 con 0x561640379e20 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4ee84e740 10 client.0 _ll_register_callbacks cb 0x7fc4d80f48e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4e3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561640374cb0 0x561640381a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4e3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561640374cb0 0x561640381a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43856/0 (socket says 172.21.15.102:43856) 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4e3e00640 1 -- 172.21.15.102:0/1587367203 learned_addr learned my addr 172.21.15.102:0/1587367203 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4e9000640 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5616403742c0 0x5616403814d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4e9a00640 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 0x561640384a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4e3e00640 1 -- 172.21.15.102:0/1587367203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 msgr2=0x561640384a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 0x561640384a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4e3e00640 1 -- 172.21.15.102:0/1587367203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5616403742c0 msgr2=0x5616403814d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5616403742c0 0x5616403814d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.088+0000 7fc4e3e00640 1 -- 172.21.15.102:0/1587367203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561640385020 con 0x561640374cb0 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.089+0000 7fc4e3e00640 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561640374cb0 0x561640381a10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x561640370bf0 tx=0x7fc4d00096b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ce1ae44d57c435 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.089+0000 7fc4caa00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.089+0000 7fc4caa00640 1 -- 172.21.15.102:0/1587367203 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d001e040 con 0x561640374cb0 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.089+0000 7fc4caa00640 1 -- 172.21.15.102:0/1587367203 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc4d00052b0 con 0x561640374cb0 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.089+0000 7fc4ee84e740 1 -- 172.21.15.102:0/1587367203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4d80c1260 con 0x561640374cb0 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.089+0000 7fc4caa00640 1 -- 172.21.15.102:0/1587367203 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc4d003d050 con 0x561640374cb0 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.089+0000 7fc4ee84e740 10 client.15306 fetch_fsmap learned FSMap version 27 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.089+0000 7fc4ee84e740 1 -- 172.21.15.102:0/1587367203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5616403f1440 con 0x561640374cb0 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.090+0000 7fc4caa00640 1 -- 172.21.15.102:0/1587367203 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4d80c1260 con 0x561640374cb0 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.090+0000 7fc4caa00640 1 -- 172.21.15.102:0/1587367203 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc4d0034430 con 0x561640374cb0 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.090+0000 7fc4ee84e740 10 client.15306 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.090+0000 7fc4ee84e740 10 client.15306 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.090+0000 7fc4ee84e740 20 client.15306 populate_metadata read hostname 'smithi102' 2024-04-26T20:35:09.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.090+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x5616403f5de0 0x5616403f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.090+0000 7fc4ee84e740 4 client.15306 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.090+0000 7fc4ee84e740 1 -- 172.21.15.102:0/1587367203 --> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] -- command(tid 0: {"prefix": "status"}) -- 0x5616402e0080 con 0x5616403f5de0 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.090+0000 7fc4e9000640 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x5616403f5de0 0x5616403f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4e9000640 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x5616403f5de0 0x5616403f4af0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f4582335bb829c62 server_cookie=7c2263367ab758d4 in_seq=0 out_seq=0 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4caa00640 10 client.15306 ms_handle_connect on v2:172.21.15.102:6838/3473487681 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4caa00640 1 -- 172.21.15.102:0/1587367203 <== mds.1 v2:172.21.15.102:6838/3473487681 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5616402e0080 con 0x5616403f5de0 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4caa00640 10 client.15306 handle_command_reply: tid=0 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 client.15306 shutdown 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 -- 172.21.15.102:0/1587367203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561640374cb0 msgr2=0x561640381a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561640374cb0 0x561640381a10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x561640370bf0 tx=0x7fc4d00096b0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 -- 172.21.15.102:0/1587367203 shutdown_connections 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.102:6838/3473487681,v1:172.21.15.102:6839/3473487681] conn(0x5616403f5de0 0x5616403f4af0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561640374cb0 0x561640381a10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561640379e20 0x561640384a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4caa00640 0 client.15306 ms_handle_reset on v2:172.21.15.102:6838/3473487681 2024-04-26T20:35:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 --2- 172.21.15.102:0/1587367203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5616403742c0 0x5616403814d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 -- 172.21.15.102:0/1587367203 >> 172.21.15.102:0/1587367203 conn(0x561640308d10 msgr2=0x561640389f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 -- 172.21.15.102:0/1587367203 shutdown_connections 2024-04-26T20:35:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.091+0000 7fc4ee84e740 1 -- 172.21.15.102:0/1587367203 wait complete. 2024-04-26T20:35:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ee84e740 20 client.15306 trim_cache size 0 max 16384 2024-04-26T20:35:09.115 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:35:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:35:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 1, 2024-04-26T20:35:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24565, 2024-04-26T20:35:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:35:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:35:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:35:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 137.450162009, 2024-04-26T20:35:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 27, 2024-04-26T20:35:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-04-26T20:35:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:35:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 176.30729320500001 2024-04-26T20:35:09.116 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:35:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 -- 172.21.15.102:0/1711546891 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4cc0a3c40 msgr2=0x7fc4cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4cc0a3c40 0x7fc4cc0a6100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x561640375510 tx=0x7fc4dc0096a0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 -- 172.21.15.102:0/1711546891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4e40772f0 msgr2=0x7fc4e419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4e40772f0 0x7fc4e419efc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8005ab0 tx=0x7fc4d8003b90 comp rx=0 tx=0).stop 2024-04-26T20:35:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 -- 172.21.15.102:0/1711546891 shutdown_connections 2024-04-26T20:35:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc4cc0a3c40 0x7fc4cc0a6100 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4e4078530 0x7fc4e419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4e4078ed0 0x7fc4e41a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 --2- 172.21.15.102:0/1711546891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4e40772f0 0x7fc4e419efc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 -- 172.21.15.102:0/1711546891 >> 172.21.15.102:0/1711546891 conn(0x7fc4e40fff10 msgr2=0x7fc4e4101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.092+0000 7fc4ea400640 1 -- 172.21.15.102:0/1711546891 shutdown_connections 2024-04-26T20:35:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.093+0000 7fc4ea400640 1 -- 172.21.15.102:0/1711546891 wait complete. 2024-04-26T20:35:09.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T20:35:09.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:08 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2982538165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:35:09.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:08 smithi130 ceph-mon[32704]: pgmap v403: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:35:09.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.320+0000 7f49eb200640 1 -- 172.21.15.102:0/2340478292 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec102770 msgr2=0x7f49ec102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.320+0000 7f49eb200640 1 --2- 172.21.15.102:0/2340478292 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec102770 0x7f49ec102b90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009860 tx=0x7f49dc02a340 comp rx=0 tx=0).stop 2024-04-26T20:35:09.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.320+0000 7f49eb200640 1 -- 172.21.15.102:0/2340478292 shutdown_connections 2024-04-26T20:35:09.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.320+0000 7f49eb200640 1 --2- 172.21.15.102:0/2340478292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49ec104320 0x7f49ec10abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.320+0000 7f49eb200640 1 --2- 172.21.15.102:0/2340478292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49ec103960 0x7f49ec103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.320+0000 7f49eb200640 1 --2- 172.21.15.102:0/2340478292 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec102770 0x7f49ec102b90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.320+0000 7f49eb200640 1 -- 172.21.15.102:0/2340478292 >> 172.21.15.102:0/2340478292 conn(0x7f49ec0fdce0 msgr2=0x7f49ec100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.320+0000 7f49eb200640 1 -- 172.21.15.102:0/2340478292 shutdown_connections 2024-04-26T20:35:09.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.320+0000 7f49eb200640 1 -- 172.21.15.102:0/2340478292 wait complete. 2024-04-26T20:35:09.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.321+0000 7f49eb200640 1 Processor -- start 2024-04-26T20:35:09.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.321+0000 7f49eb200640 1 -- start start 2024-04-26T20:35:09.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.321+0000 7f49eb200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec102770 0x7f49ec19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.321+0000 7f49eb200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49ec103960 0x7f49ec19b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.321+0000 7f49eb200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49ec104320 0x7f49ec1a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.321+0000 7f49eb200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec10e290 con 0x7f49ec103960 2024-04-26T20:35:09.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.321+0000 7f49eb200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec10e110 con 0x7f49ec102770 2024-04-26T20:35:09.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.321+0000 7f49eb200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec10e410 con 0x7f49ec104320 2024-04-26T20:35:09.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49e1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49ec103960 0x7f49ec19b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49ea800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49ec104320 0x7f49ec1a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49e1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49ec103960 0x7f49ec19b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43876/0 (socket says 172.21.15.102:43876) 2024-04-26T20:35:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49e1400640 1 -- 172.21.15.102:0/775430233 learned_addr learned my addr 172.21.15.102:0/775430233 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49e9e00640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec102770 0x7f49ec19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49e1400640 1 -- 172.21.15.102:0/775430233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49ec104320 msgr2=0x7f49ec1a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49e1400640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49ec104320 0x7f49ec1a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49e1400640 1 -- 172.21.15.102:0/775430233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec102770 msgr2=0x7f49ec19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49e1400640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec102770 0x7f49ec19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49e1400640 1 -- 172.21.15.102:0/775430233 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec1a2810 con 0x7f49ec103960 2024-04-26T20:35:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49ea800640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49ec104320 0x7f49ec1a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.322+0000 7f49e1400640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49ec103960 0x7f49ec19b130 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f49d400eb90 tx=0x7f49d400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.323+0000 7f49e3400640 1 -- 172.21.15.102:0/775430233 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4004d20 con 0x7f49ec103960 2024-04-26T20:35:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.323+0000 7f49eb200640 1 -- 172.21.15.102:0/775430233 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec1a2b00 con 0x7f49ec103960 2024-04-26T20:35:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.323+0000 7f49e3400640 1 -- 172.21.15.102:0/775430233 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49d400ee90 con 0x7f49ec103960 2024-04-26T20:35:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.323+0000 7f49e3400640 1 -- 172.21.15.102:0/775430233 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4010670 con 0x7f49ec103960 2024-04-26T20:35:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.323+0000 7f49eb200640 1 -- 172.21.15.102:0/775430233 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec1a3000 con 0x7f49ec103960 2024-04-26T20:35:09.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.326+0000 7f49f37df740 1 Processor -- start 2024-04-26T20:35:09.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.326+0000 7f49e3400640 1 -- 172.21.15.102:0/775430233 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f49d4005020 con 0x7f49ec103960 2024-04-26T20:35:09.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.326+0000 7f49f37df740 1 -- start start 2024-04-26T20:35:09.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.326+0000 7f49f37df740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 0x557f2cd676e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.326+0000 7f49f37df740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd67cb0 0x557f2cd6c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.326+0000 7f49f37df740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd6ce20 0x557f2cd6f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49f37df740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557f2cccb600 con 0x557f2cd67cb0 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49f37df740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f2cc68100 con 0x557f2cd6ce20 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49f37df740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557f2ccc75c0 con 0x557f2cd672c0 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49e9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd67cb0 0x557f2cd6c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49ea800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 0x557f2cd676e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49ea800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 0x557f2cd676e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:35138/0 (socket says 172.21.15.102:35138) 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49ea800640 1 -- 172.21.15.102:0/2487558971 learned_addr learned my addr 172.21.15.102:0/2487558971 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49e1400640 1 --2- 172.21.15.102:0/2487558971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd6ce20 0x557f2cd6f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49e9e00640 1 -- 172.21.15.102:0/2487558971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 msgr2=0x557f2cd676e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49e9e00640 1 --2- 172.21.15.102:0/2487558971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 0x557f2cd676e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49e9e00640 1 -- 172.21.15.102:0/2487558971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd6ce20 msgr2=0x557f2cd6f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49e9e00640 1 --2- 172.21.15.102:0/2487558971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd6ce20 0x557f2cd6f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49e9e00640 1 -- 172.21.15.102:0/2487558971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f2c9dd2d0 con 0x557f2cd67cb0 2024-04-26T20:35:09.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.327+0000 7f49e9e00640 1 --2- 172.21.15.102:0/2487558971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd67cb0 0x557f2cd6c7b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f49dc02a860 tx=0x7f49dc009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37206aca0604f83c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.328+0000 7f49e3400640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f49d00a3c40 0x7f49d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.328+0000 7f49db000640 1 -- 172.21.15.102:0/2487558971 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc041020 con 0x557f2cd67cb0 2024-04-26T20:35:09.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.328+0000 7f49e3400640 1 -- 172.21.15.102:0/775430233 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f49d40d0f20 con 0x7f49ec103960 2024-04-26T20:35:09.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.328+0000 7f49db000640 1 -- 172.21.15.102:0/2487558971 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49dc004040 con 0x557f2cd67cb0 2024-04-26T20:35:09.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.328+0000 7f49db000640 1 -- 172.21.15.102:0/2487558971 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc037040 con 0x557f2cd67cb0 2024-04-26T20:35:09.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.328+0000 7f49ea800640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f49d00a3c40 0x7f49d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49f37df740 1 -- 172.21.15.102:0/2487558971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd67cb0 msgr2=0x557f2cd6c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49f37df740 1 --2- 172.21.15.102:0/2487558971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd67cb0 0x557f2cd6c7b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f49dc02a860 tx=0x7f49dc009490 comp rx=0 tx=0).stop 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49ea800640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f49d00a3c40 0x7f49d00a6100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x557f2cd68510 tx=0x7f49e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49f37df740 1 -- 172.21.15.102:0/2487558971 shutdown_connections 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49f37df740 1 --2- 172.21.15.102:0/2487558971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd67cb0 0x557f2cd6c7b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49f37df740 1 --2- 172.21.15.102:0/2487558971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd6ce20 0x557f2cd6f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49f37df740 1 --2- 172.21.15.102:0/2487558971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 0x557f2cd676e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49f37df740 1 -- 172.21.15.102:0/2487558971 >> 172.21.15.102:0/2487558971 conn(0x557f2ccfbd10 msgr2=0x557f2cd62d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49f37df740 1 -- 172.21.15.102:0/2487558971 shutdown_connections 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49f37df740 1 -- 172.21.15.102:0/2487558971 wait complete. 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.329+0000 7f49f37df740 1 -- start start 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49f37df740 1 Processor -- start 2024-04-26T20:35:09.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49f37df740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd672c0 0x557f2cd716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49f37df740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd67cb0 0x557f2cd71bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49f37df740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 0x557f2cd78c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49f37df740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557f2cd6fd70 con 0x557f2cd6ce20 2024-04-26T20:35:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49f37df740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f2cc52f70 con 0x557f2cd672c0 2024-04-26T20:35:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49f37df740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557f2cd6fef0 con 0x557f2cd67cb0 2024-04-26T20:35:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49f37df740 10 client.0 _ll_register_callbacks cb 0x557f2ccc1480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:35:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49e9e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd672c0 0x557f2cd716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49ea800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 0x557f2cd78c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49e9e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd672c0 0x557f2cd716b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:60596/0 (socket says 172.21.15.102:60596) 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49ea800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 0x557f2cd78c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43896/0 (socket says 172.21.15.102:43896) 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.330+0000 7f49e9e00640 1 -- 172.21.15.102:0/1466570161 learned_addr learned my addr 172.21.15.102:0/1466570161 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49e1400640 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd67cb0 0x557f2cd71bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49e9e00640 1 -- 172.21.15.102:0/1466570161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd67cb0 msgr2=0x557f2cd71bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49e9e00640 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd67cb0 0x557f2cd71bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49e9e00640 1 -- 172.21.15.102:0/1466570161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 msgr2=0x557f2cd78c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49e9e00640 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 0x557f2cd78c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49e9e00640 1 -- 172.21.15.102:0/1466570161 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f2caa76b0 con 0x557f2cd672c0 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49e9e00640 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd672c0 0x557f2cd716b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009830 tx=0x7f49dc0058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ee8473dabb65b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49cf400640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-26T20:35:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49cf400640 1 -- 172.21.15.102:0/1466570161 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc041030 con 0x557f2cd672c0 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49f37df740 1 -- 172.21.15.102:0/1466570161 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49dc004040 con 0x557f2cd672c0 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.331+0000 7f49cf400640 1 -- 172.21.15.102:0/1466570161 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f49dc030550 con 0x557f2cd672c0 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.332+0000 7f49cf400640 1 -- 172.21.15.102:0/1466570161 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49dc058050 con 0x557f2cd672c0 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.332+0000 7f49f37df740 10 client.25442 fetch_fsmap learned FSMap version 27 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.332+0000 7f49f37df740 1 -- 172.21.15.102:0/1466570161 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557f2cd7bc10 con 0x557f2cd672c0 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.332+0000 7f49cf400640 1 -- 172.21.15.102:0/1466570161 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49dc004040 con 0x557f2cd672c0 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.332+0000 7f49cf400640 1 -- 172.21.15.102:0/1466570161 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f49dc03d780 con 0x557f2cd672c0 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.332+0000 7f49f37df740 10 client.25442 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.332+0000 7f49f37df740 10 client.25442 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.332+0000 7f49f37df740 20 client.25442 populate_metadata read hostname 'smithi102' 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.332+0000 7f49f37df740 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f2cd7ded0 0x557f2cd7f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.333+0000 7f49f37df740 4 client.25442 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.333+0000 7f49f37df740 1 -- 172.21.15.102:0/1466570161 --> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f2ccd9210 con 0x557f2cd7ded0 2024-04-26T20:35:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.333+0000 7f49e1400640 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f2cd7ded0 0x557f2cd7f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.333+0000 7f49e1400640 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f2cd7ded0 0x557f2cd7f340 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17dcc6665f582742 server_cookie=ede71ca18878b159 in_seq=0 out_seq=0 2024-04-26T20:35:09.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.333+0000 7f49cf400640 10 client.25442 ms_handle_connect on v2:172.21.15.112:6838/2318597180 2024-04-26T20:35:09.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49cf400640 1 -- 172.21.15.102:0/1466570161 <== mds.0 v2:172.21.15.112:6838/2318597180 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f2ccd9210 con 0x557f2cd7ded0 2024-04-26T20:35:09.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49cf400640 10 client.25442 handle_command_reply: tid=0 2024-04-26T20:35:09.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49f37df740 1 client.25442 shutdown 2024-04-26T20:35:09.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49f37df740 1 -- 172.21.15.102:0/1466570161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd672c0 msgr2=0x557f2cd716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49f37df740 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd672c0 0x557f2cd716b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009830 tx=0x7f49dc0058d0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49f37df740 1 -- 172.21.15.102:0/1466570161 shutdown_connections 2024-04-26T20:35:09.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49f37df740 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 0x557f2cd78c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49f37df740 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f2cd7ded0 0x557f2cd7f340 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49f37df740 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd67cb0 0x557f2cd71bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49f37df740 1 --2- 172.21.15.102:0/1466570161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd672c0 0x557f2cd716b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49cf400640 0 client.25442 ms_handle_reset on v2:172.21.15.112:6838/2318597180 2024-04-26T20:35:09.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49f37df740 1 -- 172.21.15.102:0/1466570161 >> 172.21.15.102:0/1466570161 conn(0x557f2ccfbd10 msgr2=0x557f2cd61f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.334+0000 7f49f37df740 1 -- 172.21.15.102:0/1466570161 shutdown_connections 2024-04-26T20:35:09.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.335+0000 7f49f37df740 1 -- 172.21.15.102:0/1466570161 wait complete. 2024-04-26T20:35:09.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.335+0000 7f49f37df740 20 client.25442 trim_cache size 0 max 16384 2024-04-26T20:35:09.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.354+0000 7f49f37df740 1 Processor -- start 2024-04-26T20:35:09.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.354+0000 7f49f37df740 1 -- start start 2024-04-26T20:35:09.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49f37df740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 0x557f2cd61de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49f37df740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 0x557f2cd78b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49f37df740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 0x557f2cd70cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49f37df740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557f2cd7d680 con 0x557f2cd6ce20 2024-04-26T20:35:09.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49f37df740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f2cd7dd50 con 0x557f2cd67cb0 2024-04-26T20:35:09.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49f37df740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557f2cd7e420 con 0x557f2cd672c0 2024-04-26T20:35:09.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49ea800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 0x557f2cd70cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49e9e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 0x557f2cd61de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49ea800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 0x557f2cd70cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43904/0 (socket says 172.21.15.102:43904) 2024-04-26T20:35:09.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49e1400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 0x557f2cd78b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49ea800640 1 -- 172.21.15.102:0/3152449157 learned_addr learned my addr 172.21.15.102:0/3152449157 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:09.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49e9e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 0x557f2cd61de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:35160/0 (socket says 172.21.15.102:35160) 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49e1400640 1 -- 172.21.15.102:0/3152449157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 msgr2=0x557f2cd61de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49e1400640 1 --2- 172.21.15.102:0/3152449157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 0x557f2cd61de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49e1400640 1 -- 172.21.15.102:0/3152449157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 msgr2=0x557f2cd70cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49e1400640 1 --2- 172.21.15.102:0/3152449157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 0x557f2cd70cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.355+0000 7f49e1400640 1 -- 172.21.15.102:0/3152449157 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f2cd7efa0 con 0x557f2cd67cb0 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.356+0000 7f49e1400640 1 --2- 172.21.15.102:0/3152449157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 0x557f2cd78b50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f49d40ce520 tx=0x7f49d40bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=551248fe72a4b2ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.356+0000 7f49da600640 1 -- 172.21.15.102:0/3152449157 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d40b5020 con 0x557f2cd67cb0 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.356+0000 7f49da600640 1 -- 172.21.15.102:0/3152449157 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49d40b6260 con 0x557f2cd67cb0 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.356+0000 7f49da600640 1 -- 172.21.15.102:0/3152449157 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d409b080 con 0x557f2cd67cb0 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 -- 172.21.15.102:0/3152449157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 msgr2=0x557f2cd78b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 --2- 172.21.15.102:0/3152449157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 0x557f2cd78b50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f49d40ce520 tx=0x7f49d40bd040 comp rx=0 tx=0).stop 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 -- 172.21.15.102:0/3152449157 shutdown_connections 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 --2- 172.21.15.102:0/3152449157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd6ce20 0x557f2cd70cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 --2- 172.21.15.102:0/3152449157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 0x557f2cd78b50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 --2- 172.21.15.102:0/3152449157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd672c0 0x557f2cd61de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 -- 172.21.15.102:0/3152449157 >> 172.21.15.102:0/3152449157 conn(0x557f2ccfbd10 msgr2=0x557f2cd616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 -- 172.21.15.102:0/3152449157 shutdown_connections 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 -- 172.21.15.102:0/3152449157 wait complete. 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 -- start start 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.357+0000 7f49f37df740 1 Processor -- start 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49f37df740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd672c0 0x557f2cd78530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49f37df740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 0x557f2cd78a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49f37df740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd6ce20 0x557f2cd74fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49f37df740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557f2cd7f1c0 con 0x557f2cd672c0 2024-04-26T20:35:09.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49f37df740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557f2cd7eaf0 con 0x557f2cd67cb0 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49f37df740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557f2cd7f890 con 0x557f2cd6ce20 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49f37df740 10 client.0 _ll_register_callbacks cb 0x7f49e4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49e9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd672c0 0x557f2cd78530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49e9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd672c0 0x557f2cd78530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43920/0 (socket says 172.21.15.102:43920) 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49e1400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 0x557f2cd78a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49e9e00640 1 -- 172.21.15.102:0/1156563522 learned_addr learned my addr 172.21.15.102:0/1156563522 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.358+0000 7f49ea800640 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd6ce20 0x557f2cd74fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49e9e00640 1 -- 172.21.15.102:0/1156563522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd6ce20 msgr2=0x557f2cd74fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49e9e00640 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd6ce20 0x557f2cd74fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49e9e00640 1 -- 172.21.15.102:0/1156563522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 msgr2=0x557f2cd78a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49e9e00640 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 0x557f2cd78a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49e9e00640 1 -- 172.21.15.102:0/1156563522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f2cd756c0 con 0x557f2cd672c0 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49e1400640 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 0x557f2cd78a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49e9e00640 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd672c0 0x557f2cd78530 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009860 tx=0x7f49dc030800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbf5763390a1cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:09.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49cf400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-26T20:35:09.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49cf400640 1 -- 172.21.15.102:0/1156563522 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc04a020 con 0x557f2cd672c0 2024-04-26T20:35:09.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49cf400640 1 -- 172.21.15.102:0/1156563522 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f49dc03d690 con 0x557f2cd672c0 2024-04-26T20:35:09.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49f37df740 1 -- 172.21.15.102:0/1156563522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49d40b6260 con 0x557f2cd672c0 2024-04-26T20:35:09.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49cf400640 1 -- 172.21.15.102:0/1156563522 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49dc041040 con 0x557f2cd672c0 2024-04-26T20:35:09.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49f37df740 10 client.15324 fetch_fsmap learned FSMap version 27 2024-04-26T20:35:09.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.359+0000 7f49f37df740 1 -- 172.21.15.102:0/1156563522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557f2cde4a80 con 0x557f2cd672c0 2024-04-26T20:35:09.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.360+0000 7f49cf400640 1 -- 172.21.15.102:0/1156563522 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49d40b6260 con 0x557f2cd672c0 2024-04-26T20:35:09.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.360+0000 7f49cf400640 1 -- 172.21.15.102:0/1156563522 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f49dc032440 con 0x557f2cd672c0 2024-04-26T20:35:09.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.360+0000 7f49f37df740 10 client.15324 fetch_fsmap finished waiting for FSMap version 27 2024-04-26T20:35:09.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.360+0000 7f49f37df740 10 client.15324 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T20:35:09.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.360+0000 7f49f37df740 20 client.15324 populate_metadata read hostname 'smithi102' 2024-04-26T20:35:09.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.360+0000 7f49f37df740 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f2cde9690 0x557f2cde83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:09.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.360+0000 7f49f37df740 4 client.15324 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T20:35:09.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.360+0000 7f49f37df740 1 -- 172.21.15.102:0/1156563522 --> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] -- command(tid 0: {"prefix": "status"}) -- 0x557f2ccd3080 con 0x557f2cde9690 2024-04-26T20:35:09.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.361+0000 7f49e1400640 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f2cde9690 0x557f2cde83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:09.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.361+0000 7f49e1400640 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f2cde9690 0x557f2cde83a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f17039cc381ca85 server_cookie=7f4b34b6c7b8acb7 in_seq=0 out_seq=0 2024-04-26T20:35:09.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.361+0000 7f49cf400640 10 client.15324 ms_handle_connect on v2:172.21.15.112:6838/2318597180 2024-04-26T20:35:09.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.361+0000 7f49cf400640 1 -- 172.21.15.102:0/1156563522 <== mds.0 v2:172.21.15.112:6838/2318597180 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557f2ccd3080 con 0x557f2cde9690 2024-04-26T20:35:09.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.361+0000 7f49cf400640 10 client.15324 handle_command_reply: tid=0 2024-04-26T20:35:09.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.361+0000 7f49f37df740 1 client.15324 shutdown 2024-04-26T20:35:09.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.361+0000 7f49f37df740 1 -- 172.21.15.102:0/1156563522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd672c0 msgr2=0x557f2cd78530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.361+0000 7f49f37df740 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd672c0 0x557f2cd78530 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009860 tx=0x7f49dc030800 comp rx=0 tx=0).stop 2024-04-26T20:35:09.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.362+0000 7f49f37df740 1 -- 172.21.15.102:0/1156563522 shutdown_connections 2024-04-26T20:35:09.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.362+0000 7f49f37df740 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.112:6838/2318597180,v1:172.21.15.112:6839/2318597180] conn(0x557f2cde9690 0x557f2cde83a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.362+0000 7f49f37df740 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557f2cd6ce20 0x557f2cd74fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.362+0000 7f49f37df740 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f2cd67cb0 0x557f2cd78a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.362+0000 7f49f37df740 1 --2- 172.21.15.102:0/1156563522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557f2cd672c0 0x557f2cd78530 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.362+0000 7f49cf400640 0 client.15324 ms_handle_reset on v2:172.21.15.112:6838/2318597180 2024-04-26T20:35:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.362+0000 7f49f37df740 1 -- 172.21.15.102:0/1156563522 >> 172.21.15.102:0/1156563522 conn(0x557f2ccfbd10 msgr2=0x557f2cd61f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.362+0000 7f49f37df740 1 -- 172.21.15.102:0/1156563522 shutdown_connections 2024-04-26T20:35:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.362+0000 7f49f37df740 1 -- 172.21.15.102:0/1156563522 wait complete. 2024-04-26T20:35:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.362+0000 7f49f37df740 20 client.15324 trim_cache size 0 max 16384 2024-04-26T20:35:09.374 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-26T20:35:09.374 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "a94c830a-040a-11ef-bc93-c7b262605968", 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24593, 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 163.05500907300001, 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 27, 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 169.412742385 2024-04-26T20:35:09.375 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-26T20:35:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 -- 172.21.15.102:0/775430233 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f49d00a3c40 msgr2=0x7f49d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f49d00a3c40 0x7f49d00a6100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x557f2cd68510 tx=0x7f49e4009260 comp rx=0 tx=0).stop 2024-04-26T20:35:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 -- 172.21.15.102:0/775430233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49ec103960 msgr2=0x7f49ec19b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49ec103960 0x7f49ec19b130 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f49d400eb90 tx=0x7f49d400c6a0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 -- 172.21.15.102:0/775430233 shutdown_connections 2024-04-26T20:35:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f49d00a3c40 0x7f49d00a6100 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49ec103960 0x7f49ec19b130 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49ec104320 0x7f49ec1a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 --2- 172.21.15.102:0/775430233 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec102770 0x7f49ec19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:09.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 -- 172.21.15.102:0/775430233 >> 172.21.15.102:0/775430233 conn(0x7f49ec0fdce0 msgr2=0x7f49ec109130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:09.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 -- 172.21.15.102:0/775430233 shutdown_connections 2024-04-26T20:35:09.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:09.363+0000 7f49eb200640 1 -- 172.21.15.102:0/775430233 wait complete. 2024-04-26T20:35:09.381 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T20:35:09.395 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T20:35:09.395 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T20:35:09.395 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T20:35:09.395 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T20:35:09.411 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T20:35:09.411 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T20:35:09.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:35:09.412 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-04-26T20:35:09.466 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-26T20:35:09.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:35:09.467 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T20:35:09.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:35:09.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-04-26T20:35:09.641 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-26T20:35:09.642 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T20:35:09.642 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T20:35:09.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid a94c830a-040a-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T20:35:10.071 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T20:35:11.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:11 smithi102 ceph-mon[28094]: pgmap v404: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:35:12.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:11 smithi112 ceph-mon[32769]: pgmap v404: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:35:12.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:11 smithi130 ceph-mon[32704]: pgmap v404: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:35:12.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.475+0000 7f8946c00700 1 -- 172.21.15.102:0/4177081738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8940106000 msgr2=0x7f8940112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.475+0000 7f8946c00700 1 --2- 172.21.15.102:0/4177081738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8940106000 0x7f8940112510 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f893800b0a0 tx=0x7f893800b3b0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.475+0000 7f8946c00700 1 -- 172.21.15.102:0/4177081738 shutdown_connections 2024-04-26T20:35:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.475+0000 7f8946c00700 1 --2- 172.21.15.102:0/4177081738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8940106000 0x7f8940112510 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.475+0000 7f8946c00700 1 --2- 172.21.15.102:0/4177081738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f894010b640 0x7f894010ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.475+0000 7f8946c00700 1 --2- 172.21.15.102:0/4177081738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8940105640 0x7f8940105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.475+0000 7f8946c00700 1 -- 172.21.15.102:0/4177081738 >> 172.21.15.102:0/4177081738 conn(0x7f8940101130 msgr2=0x7f8940103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.475+0000 7f8946c00700 1 -- 172.21.15.102:0/4177081738 shutdown_connections 2024-04-26T20:35:12.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.475+0000 7f8946c00700 1 -- 172.21.15.102:0/4177081738 wait complete. 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.476+0000 7f8946c00700 1 Processor -- start 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.476+0000 7f8946c00700 1 -- start start 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f8946c00700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8940105640 0x7f89401a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f8946c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8940106000 0x7f89401a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f8946c00700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f894010b640 0x7f894019bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f8946c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8940115890 con 0x7f8940106000 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f8946c00700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8940115710 con 0x7f894010b640 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f8946c00700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8940115a10 con 0x7f8940105640 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f893f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8940106000 0x7f89401a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f893f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8940106000 0x7f89401a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43928/0 (socket says 172.21.15.102:43928) 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f893f400700 1 -- 172.21.15.102:0/3122565961 learned_addr learned my addr 172.21.15.102:0/3122565961 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f893fe00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8940105640 0x7f89401a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.477+0000 7f8944c00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f894010b640 0x7f894019bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:12.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f893f400700 1 -- 172.21.15.102:0/3122565961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8940105640 msgr2=0x7f89401a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f893f400700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8940105640 0x7f89401a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f893f400700 1 -- 172.21.15.102:0/3122565961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f894010b640 msgr2=0x7f894019bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f893f400700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f894010b640 0x7f894019bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f893f400700 1 -- 172.21.15.102:0/3122565961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894019c2b0 con 0x7f8940106000 2024-04-26T20:35:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f8944c00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f894010b640 0x7f894019bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:35:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f893f400700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8940106000 0x7f89401a1e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f892c00e9a0 tx=0x7f892c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f893cc00700 1 -- 172.21.15.102:0/3122565961 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c00ca10 con 0x7f8940106000 2024-04-26T20:35:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f893cc00700 1 -- 172.21.15.102:0/3122565961 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f892c00cbb0 con 0x7f8940106000 2024-04-26T20:35:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f8946c00700 1 -- 172.21.15.102:0/3122565961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f894019c5a0 con 0x7f8940106000 2024-04-26T20:35:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f893cc00700 1 -- 172.21.15.102:0/3122565961 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c018680 con 0x7f8940106000 2024-04-26T20:35:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.478+0000 7f8946c00700 1 -- 172.21.15.102:0/3122565961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89401a8810 con 0x7f8940106000 2024-04-26T20:35:12.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.479+0000 7f8946c00700 1 -- 172.21.15.102:0/3122565961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f894006a110 con 0x7f8940106000 2024-04-26T20:35:12.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.482+0000 7f893cc00700 1 -- 172.21.15.102:0/3122565961 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f892c010430 con 0x7f8940106000 2024-04-26T20:35:12.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.484+0000 7f893cc00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f89240a3cd0 0x7f89240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:12.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.485+0000 7f893cc00700 1 -- 172.21.15.102:0/3122565961 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f892c0d0e40 con 0x7f8940106000 2024-04-26T20:35:12.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.485+0000 7f893fe00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f89240a3cd0 0x7f89240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:12.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.486+0000 7f893fe00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f89240a3cd0 0x7f89240a6190 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f89300045b0 tx=0x7f893000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.487+0000 7f893cc00700 1 -- 172.21.15.102:0/3122565961 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f892c09d2b0 con 0x7f8940106000 2024-04-26T20:35:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.765+0000 7f8946c00700 1 -- 172.21.15.102:0/3122565961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f894002e490 con 0x7f8940106000 2024-04-26T20:35:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.765+0000 7f893cc00700 1 -- 172.21.15.102:0/3122565961 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7f892c09cbb0 con 0x7f8940106000 2024-04-26T20:35:12.766 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T20:35:12.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.768+0000 7f8946c00700 1 -- 172.21.15.102:0/3122565961 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f89240a3cd0 msgr2=0x7f89240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:12.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.768+0000 7f8946c00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f89240a3cd0 0x7f89240a6190 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f89300045b0 tx=0x7f893000b3c0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.769+0000 7f8946c00700 1 -- 172.21.15.102:0/3122565961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8940106000 msgr2=0x7f89401a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:12.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.769+0000 7f8946c00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8940106000 0x7f89401a1e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f892c00e9a0 tx=0x7f892c00ecb0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.769+0000 7f8946c00700 1 -- 172.21.15.102:0/3122565961 shutdown_connections 2024-04-26T20:35:12.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.769+0000 7f8946c00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f89240a3cd0 0x7f89240a6190 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.769+0000 7f8946c00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f894010b640 0x7f894019bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.769+0000 7f8946c00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8940105640 0x7f89401a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.769+0000 7f8946c00700 1 --2- 172.21.15.102:0/3122565961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8940106000 0x7f89401a1e80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:12.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.769+0000 7f8946c00700 1 -- 172.21.15.102:0/3122565961 >> 172.21.15.102:0/3122565961 conn(0x7f8940101130 msgr2=0x7f89401017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:12.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.769+0000 7f8946c00700 1 -- 172.21.15.102:0/3122565961 shutdown_connections 2024-04-26T20:35:12.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:12.769+0000 7f8946c00700 1 -- 172.21.15.102:0/3122565961 wait complete. 2024-04-26T20:35:13.682 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T20:35:13.682 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T20:35:13.682 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-04-26T20:35:13.682 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T20:35:13.682 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T20:35:13.682 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T20:35:13.682 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T20:35:13.682 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T20:35:13.683 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T20:35:13.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:35:13.683 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-04-26T20:35:13.740 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T20:35:13.740 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T20:35:13.741 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-04-26T20:35:13.742 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-26T20:35:13.742 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 41939sec preferred_lft 41939sec 2024-04-26T20:35:13.742 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-04-26T20:35:13.742 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-04-26T20:35:13.743 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T20:35:13.743 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:35:13.743 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-26T20:35:13.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add name ceph-brx type bridge 2024-04-26T20:35:13.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr flush dev ceph-brx 2024-04-26T20:35:13.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set ceph-brx up 2024-04-26T20:35:13.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T20:35:13.743 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-26T20:35:13.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:35:13.873 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:13 smithi102 ceph-mon[28094]: pgmap v405: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:35:13.874 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:13 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3122565961' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:35:13.924 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:13 smithi130 ceph-mon[32704]: pgmap v405: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:35:13.924 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:13 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3122565961' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:35:13.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:35:13.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:35:13.995 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T20:35:14.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:13 smithi112 ceph-mon[32769]: pgmap v405: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:35:14.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:13 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3122565961' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T20:35:14.057 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-04-26T20:35:14.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:35:14.061 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-04-26T20:35:14.102 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-04-26T20:35:14.103 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T20:35:14.103 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-26T20:35:14.103 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-26T20:35:14.103 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T20:35:14.103 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:35:14.103 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-26T20:35:14.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-26T20:35:14.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-26T20:35:14.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-26T20:35:14.103 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-26T20:35:14.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:35:14.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:35:14.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:35:14.356 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-04-26T20:35:14.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:35:14.399 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2024-04-26T20:35:14.491 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:35:14.491 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-26T20:35:14.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T20:35:14.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-26T20:35:14.491 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-26T20:35:14.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:35:14.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:35:14.668 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-26T20:35:14.668 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:35:14.668 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-26T20:35:14.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-26T20:35:14.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T20:35:14.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-26T20:35:14.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-26T20:35:14.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-26T20:35:14.668 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-26T20:35:14.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:35:14.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:35:14.924 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T20:35:14.924 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-26T20:35:14.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-04-26T20:35:14.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T20:35:14.924 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-26T20:35:15.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:35:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:35:15.121 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T20:35:15.121 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T20:35:15.161 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T20:35:15.161 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T20:35:15.161 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T20:35:15.248 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T20:35:15.248 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T20:35:15.248 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/7baa4043-cba2-4147-ad16-9a164859bb63 2024-04-26T20:35:15.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:35:15.249 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/7baa4043-cba2-4147-ad16-9a164859bb63 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-04-26T20:35:15.407 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-04-26T20:35:15.407 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-04-26T20:35:16.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:15 smithi112 ceph-mon[32769]: pgmap v406: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:35:16.095 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:15 smithi102 ceph-mon[28094]: pgmap v406: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:35:16.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:15 smithi130 ceph-mon[32704]: pgmap v406: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:35:16.123 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-04-26T20:35:16.123 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-04-26T20:35:16.123 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.112:3300,172.21.15.130:3300" 2024-04-26T20:35:16.123 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with new device syntax: 0@a94c830a-040a-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/7baa4043-cba2-4147-ad16-9a164859bb63 2024-04-26T20:35:16.124 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.102:3300/172.21.15.112:3300/172.21.15.130:3300" will pass to kernel 2024-04-26T20:35:16.127 INFO:teuthology.orchestra.run.smithi102.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T20:35:16.127 INFO:teuthology.orchestra.run.smithi102.stdout: You just mounted a file system that supports labels which does not 2024-04-26T20:35:16.127 INFO:teuthology.orchestra.run.smithi102.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T20:35:16.128 INFO:teuthology.orchestra.run.smithi102.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T20:35:16.128 INFO:teuthology.orchestra.run.smithi102.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T20:35:16.132 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T20:35:16.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:35:16.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T20:35:16.215 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T20:35:16.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T20:35:16.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:35:16.336 INFO:teuthology.orchestra.run.smithi102.stdout:client15333 2024-04-26T20:35:16.336 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:35:16.336 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T20:35:16.380 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T20:35:16.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T20:35:16.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:35:16.522 INFO:teuthology.orchestra.run.smithi102.stdout:client15333 2024-04-26T20:35:16.523 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T20:35:16.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T20:35:16.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:35:16.628 INFO:teuthology.orchestra.run.smithi102.stdout:client15333 2024-04-26T20:35:16.628 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/a94c830a-040a-11ef-bc93-c7b262605968.client15333/status') 2024-04-26T20:35:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T20:35:16.756 INFO:teuthology.orchestra.run.smithi102.stdout:instance: client.15333 (3)192.168.0.1:0/4291611369 2024-04-26T20:35:16.756 INFO:teuthology.orchestra.run.smithi102.stdout:blocklisted: false 2024-04-26T20:35:16.756 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records in 2024-04-26T20:35:16.756 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records out 2024-04-26T20:35:16.756 INFO:teuthology.orchestra.run.smithi102.stderr:70 bytes copied, 7.7194e-05 s, 907 kB/s 2024-04-26T20:35:16.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T20:35:16.768 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T20:35:16.785 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T20:35:16.785 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-04-26T20:35:16.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-26T20:35:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.975+0000 7fa007200640 1 -- 172.21.15.102:0/1802788507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0000768f0 msgr2=0x7fa000074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.975+0000 7fa007200640 1 --2- 172.21.15.102:0/1802788507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0000768f0 0x7fa000074d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0009900 tx=0x7f9ff002a490 comp rx=0 tx=0).stop 2024-04-26T20:35:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.976+0000 7fa007200640 1 -- 172.21.15.102:0/1802788507 shutdown_connections 2024-04-26T20:35:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.976+0000 7fa007200640 1 --2- 172.21.15.102:0/1802788507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa00010a840 0x7fa00010ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.976+0000 7fa007200640 1 --2- 172.21.15.102:0/1802788507 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa000075310 0x7fa000075770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.976+0000 7fa007200640 1 --2- 172.21.15.102:0/1802788507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0000768f0 0x7fa000074d40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.976+0000 7fa007200640 1 -- 172.21.15.102:0/1802788507 >> 172.21.15.102:0/1802788507 conn(0x7fa0000fdf20 msgr2=0x7fa000100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.976+0000 7fa007200640 1 -- 172.21.15.102:0/1802788507 shutdown_connections 2024-04-26T20:35:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.976+0000 7fa007200640 1 -- 172.21.15.102:0/1802788507 wait complete. 2024-04-26T20:35:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7fa007200640 1 Processor -- start 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7fa007200640 1 -- start start 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7fa007200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa000075310 0x7fa00019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7fa007200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0000768f0 0x7fa00019b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7fa007200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa00010a840 0x7fa0001a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7fa007200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa000110710 con 0x7fa0000768f0 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7fa007200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa000110590 con 0x7fa000075310 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7fa007200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa000110890 con 0x7fa00010a840 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7f9fffe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa000075310 0x7fa00019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7f9fffe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa000075310 0x7fa00019acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:42556/0 (socket says 172.21.15.102:42556) 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7f9fffe00640 1 -- 172.21.15.102:0/1682399751 learned_addr learned my addr 172.21.15.102:0/1682399751 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7f9fff400640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0000768f0 0x7fa00019b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.977+0000 7fa005200640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa00010a840 0x7fa0001a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.978+0000 7f9fffe00640 1 -- 172.21.15.102:0/1682399751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa00010a840 msgr2=0x7fa0001a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:16.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.978+0000 7f9fffe00640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa00010a840 0x7fa0001a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:16.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.978+0000 7f9fffe00640 1 -- 172.21.15.102:0/1682399751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0000768f0 msgr2=0x7fa00019b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:16.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.978+0000 7f9fffe00640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0000768f0 0x7fa00019b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:16.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.978+0000 7f9fffe00640 1 -- 172.21.15.102:0/1682399751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0001a2900 con 0x7fa000075310 2024-04-26T20:35:16.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.978+0000 7f9fffe00640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa000075310 0x7fa00019acc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0005ab0 tx=0x7f9ff0003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:16.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.978+0000 7f9ffcc00640 1 -- 172.21.15.102:0/1682399751 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff003f070 con 0x7fa000075310 2024-04-26T20:35:16.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.978+0000 7f9ffcc00640 1 -- 172.21.15.102:0/1682399751 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ff0031c50 con 0x7fa000075310 2024-04-26T20:35:16.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.978+0000 7fa007200640 1 -- 172.21.15.102:0/1682399751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0001a2b90 con 0x7fa000075310 2024-04-26T20:35:16.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.978+0000 7f9ffcc00640 1 -- 172.21.15.102:0/1682399751 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0035030 con 0x7fa000075310 2024-04-26T20:35:16.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.979+0000 7fa007200640 1 -- 172.21.15.102:0/1682399751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0001a3080 con 0x7fa000075310 2024-04-26T20:35:16.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.980+0000 7fa007200640 1 -- 172.21.15.102:0/1682399751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa000109600 con 0x7fa000075310 2024-04-26T20:35:16.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.981+0000 7f9ffcc00640 1 -- 172.21.15.102:0/1682399751 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f9ff0042070 con 0x7fa000075310 2024-04-26T20:35:16.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.983+0000 7f9ffcc00640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9fe40a3cf0 0x7f9fe40a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:16.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.983+0000 7f9ffcc00640 1 -- 172.21.15.102:0/1682399751 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f9ff00f10d0 con 0x7fa000075310 2024-04-26T20:35:16.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.983+0000 7f9fff400640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9fe40a3cf0 0x7f9fe40a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:16.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.984+0000 7f9fff400640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9fe40a3cf0 0x7f9fe40a61b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0095f0 tx=0x7f9fec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:16.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:16.987+0000 7f9ffcc00640 1 -- 172.21.15.102:0/1682399751 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9ff00bcef0 con 0x7fa000075310 2024-04-26T20:35:17.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:16 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:35:17.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:16 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:35:17.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:16 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:35:17.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:17.235+0000 7fa007200640 1 -- 172.21.15.102:0/1682399751 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fa00002dcd0 con 0x7fa000075310 2024-04-26T20:35:17.928 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:17 smithi112 ceph-mon[32769]: pgmap v407: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:35:17.928 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:17.928 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:17.928 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:17.928 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:17.928 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:17.929 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:17 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:17.929 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:17 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1682399751' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T20:35:17.929 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:17 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T20:35:18.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:17 smithi130 ceph-mon[32704]: pgmap v407: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:35:18.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:17 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:17 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1682399751' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T20:35:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:17 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T20:35:18.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.124+0000 7f9ffcc00640 1 -- 172.21.15.102:0/1682399751 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) ==== 118+0+0 (secure 0 0 0) 0x7f9ff00c11f0 con 0x7fa000075310 2024-04-26T20:35:18.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.127+0000 7fa007200640 1 -- 172.21.15.102:0/1682399751 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9fe40a3cf0 msgr2=0x7f9fe40a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:18.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.127+0000 7fa007200640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9fe40a3cf0 0x7f9fe40a61b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0095f0 tx=0x7f9fec005f90 comp rx=0 tx=0).stop 2024-04-26T20:35:18.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.127+0000 7fa007200640 1 -- 172.21.15.102:0/1682399751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa000075310 msgr2=0x7fa00019acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:18.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.127+0000 7fa007200640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa000075310 0x7fa00019acc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0005ab0 tx=0x7f9ff0003b90 comp rx=0 tx=0).stop 2024-04-26T20:35:18.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.128+0000 7fa007200640 1 -- 172.21.15.102:0/1682399751 shutdown_connections 2024-04-26T20:35:18.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.128+0000 7fa007200640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9fe40a3cf0 0x7f9fe40a61b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:18.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.128+0000 7fa007200640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0000768f0 0x7fa00019b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:18.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.128+0000 7fa007200640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa00010a840 0x7fa0001a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:18.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.128+0000 7fa007200640 1 --2- 172.21.15.102:0/1682399751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa000075310 0x7fa00019acc0 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0005ab0 tx=0x7f9ff0003b90 comp rx=0 tx=0).stop 2024-04-26T20:35:18.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.128+0000 7fa007200640 1 -- 172.21.15.102:0/1682399751 >> 172.21.15.102:0/1682399751 conn(0x7fa0000fdf20 msgr2=0x7fa0001067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:18.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.128+0000 7fa007200640 1 -- 172.21.15.102:0/1682399751 shutdown_connections 2024-04-26T20:35:18.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.128+0000 7fa007200640 1 -- 172.21.15.102:0/1682399751 wait complete. 2024-04-26T20:35:18.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:17 smithi102 ceph-mon[28094]: pgmap v407: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:35:18.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:17 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:18.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:17 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1682399751' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T20:35:18.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:17 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T20:35:18.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-26T20:35:18.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.329+0000 7f806ac00640 1 -- 172.21.15.102:0/3610677533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80641016d0 msgr2=0x7f8064101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:18.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.329+0000 7f806ac00640 1 --2- 172.21.15.102:0/3610677533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80641016d0 0x7f8064101af0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8058009900 tx=0x7f805802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:35:18.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.329+0000 7f806ac00640 1 -- 172.21.15.102:0/3610677533 shutdown_connections 2024-04-26T20:35:18.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.329+0000 7f806ac00640 1 --2- 172.21.15.102:0/3610677533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8064103280 0x7f8064109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:18.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.329+0000 7f806ac00640 1 --2- 172.21.15.102:0/3610677533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80641028c0 0x7f8064102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:18.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.329+0000 7f806ac00640 1 --2- 172.21.15.102:0/3610677533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80641016d0 0x7f8064101af0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:18.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.329+0000 7f806ac00640 1 -- 172.21.15.102:0/3610677533 >> 172.21.15.102:0/3610677533 conn(0x7f80640fcc40 msgr2=0x7f80640ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:35:18.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.329+0000 7f806ac00640 1 -- 172.21.15.102:0/3610677533 shutdown_connections 2024-04-26T20:35:18.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.329+0000 7f806ac00640 1 -- 172.21.15.102:0/3610677533 wait complete. 2024-04-26T20:35:18.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.330+0000 7f806ac00640 1 Processor -- start 2024-04-26T20:35:18.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.330+0000 7f806ac00640 1 -- start start 2024-04-26T20:35:18.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.330+0000 7f806ac00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80641016d0 0x7f806419ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:18.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.330+0000 7f806ac00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80641028c0 0x7f806419b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:18.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.330+0000 7f806ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8064103280 0x7f80641a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:18.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.330+0000 7f806ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f806410e340 con 0x7f8064103280 2024-04-26T20:35:18.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.330+0000 7f806ac00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f806410e1c0 con 0x7f80641016d0 2024-04-26T20:35:18.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.330+0000 7f806ac00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f806410e4c0 con 0x7f80641028c0 2024-04-26T20:35:18.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f8068e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80641028c0 0x7f806419b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f8068e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80641028c0 0x7f806419b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:52424/0 (socket says 172.21.15.102:52424) 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f8068e00640 1 -- 172.21.15.102:0/806585123 learned_addr learned my addr 172.21.15.102:0/806585123 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f806a200640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8064103280 0x7f80641a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f8069800640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80641016d0 0x7f806419ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f8068e00640 1 -- 172.21.15.102:0/806585123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80641016d0 msgr2=0x7f806419ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f8068e00640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80641016d0 0x7f806419ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f8068e00640 1 -- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8064103280 msgr2=0x7f80641a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f8068e00640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8064103280 0x7f80641a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f8068e00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80641a2830 con 0x7f80641028c0 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.331+0000 7f8068e00640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80641028c0 0x7f806419b160 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f805000e9f0 tx=0x7f805000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.332+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805000cd80 con 0x7f80641028c0 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.332+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80500108e0 con 0x7f80641028c0 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.332+0000 7f806ac00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80641a2b20 con 0x7f80641028c0 2024-04-26T20:35:18.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.332+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050018980 con 0x7f80641028c0 2024-04-26T20:35:18.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.332+0000 7f806ac00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80641a3040 con 0x7f80641028c0 2024-04-26T20:35:18.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.333+0000 7f806ac00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80641a3210 con 0x7f80641028c0 2024-04-26T20:35:18.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.335+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8050010430 con 0x7f80641028c0 2024-04-26T20:35:18.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.336+0000 7f8062000640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f804c0a3c30 0x7f804c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:35:18.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.337+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f80500d0ca0 con 0x7f80641028c0 2024-04-26T20:35:18.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.337+0000 7f8069800640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f804c0a3c30 0x7f804c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:35:18.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.337+0000 7f8069800640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f804c0a3c30 0x7f804c0a60f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8058005ab0 tx=0x7f8058005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:35:18.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.341+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f805009cb20 con 0x7f80641028c0 2024-04-26T20:35:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:35:18.591+0000 7f806ac00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f806410bbf0 con 0x7f804c0a3c30 2024-04-26T20:35:19.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:19 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T20:35:19.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:19 smithi102 ceph-mon[28094]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T20:35:19.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:19 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:19 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:19 smithi102 ceph-mon[28094]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:35:19.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:19 smithi102 ceph-mon[28094]: pgmap v408: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-26T20:35:19.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:19 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:19 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:19 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T20:35:19.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:19 smithi112 ceph-mon[32769]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T20:35:19.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:19 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:19 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:19 smithi112 ceph-mon[32769]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:35:19.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:19 smithi112 ceph-mon[32769]: pgmap v408: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-26T20:35:19.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:19 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:19 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:19 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T20:35:19.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:19 smithi130 ceph-mon[32704]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-26T20:35:19.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:19 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:19 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:19 smithi130 ceph-mon[32704]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:35:19.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:19 smithi130 ceph-mon[32704]: pgmap v408: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-26T20:35:19.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:19 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:19.623 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:19 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:21.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:20 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:21.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:20 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:21.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:20 smithi112 ceph-mon[32769]: pgmap v409: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-26T20:35:21.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:20 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:21.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:20 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:21.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:20 smithi130 ceph-mon[32704]: pgmap v409: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-26T20:35:21.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:20 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:21.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:20 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:21.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:20 smithi102 ceph-mon[28094]: pgmap v409: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-26T20:35:22.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:35:22.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:35:22.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:22.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:35:22.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:35:22.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:22.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:35:22.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:35:22.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:35:23.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:23 smithi112 ceph-mon[32769]: pgmap v410: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-26T20:35:23.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:23 smithi130 ceph-mon[32704]: pgmap v410: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-26T20:35:23.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:23 smithi102 ceph-mon[28094]: pgmap v410: 129 pgs: 129 active+clean; 637 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-26T20:35:26.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:25 smithi112 ceph-mon[32769]: pgmap v411: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 852 B/s wr, 12 op/s 2024-04-26T20:35:26.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:25 smithi130 ceph-mon[32704]: pgmap v411: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 852 B/s wr, 12 op/s 2024-04-26T20:35:26.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:25 smithi102 ceph-mon[28094]: pgmap v411: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 852 B/s wr, 12 op/s 2024-04-26T20:35:28.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:27 smithi112 ceph-mon[32769]: pgmap v412: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 9 op/s 2024-04-26T20:35:28.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:27 smithi130 ceph-mon[32704]: pgmap v412: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 9 op/s 2024-04-26T20:35:28.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:27 smithi102 ceph-mon[28094]: pgmap v412: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 9 op/s 2024-04-26T20:35:30.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:29 smithi112 ceph-mon[32769]: pgmap v413: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 852 B/s wr, 11 op/s 2024-04-26T20:35:30.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:29 smithi130 ceph-mon[32704]: pgmap v413: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 852 B/s wr, 11 op/s 2024-04-26T20:35:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:29 smithi102 ceph-mon[28094]: pgmap v413: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 852 B/s wr, 11 op/s 2024-04-26T20:35:32.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:31 smithi112 ceph-mon[32769]: pgmap v414: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-26T20:35:32.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:31 smithi130 ceph-mon[32704]: pgmap v414: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-26T20:35:32.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:31 smithi102 ceph-mon[28094]: pgmap v414: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-26T20:35:34.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:33 smithi112 ceph-mon[32769]: pgmap v415: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T20:35:34.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:33 smithi130 ceph-mon[32704]: pgmap v415: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T20:35:34.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:33 smithi102 ceph-mon[28094]: pgmap v415: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-26T20:35:36.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:35 smithi112 ceph-mon[32769]: pgmap v416: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-26T20:35:36.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:35 smithi130 ceph-mon[32704]: pgmap v416: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-26T20:35:36.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:35 smithi102 ceph-mon[28094]: pgmap v416: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-26T20:35:37.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:37 smithi102 ceph-mon[28094]: pgmap v417: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:35:38.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:37 smithi112 ceph-mon[32769]: pgmap v417: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:35:38.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:37 smithi130 ceph-mon[32704]: pgmap v417: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:35:40.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:39 smithi112 ceph-mon[32769]: pgmap v418: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:35:40.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:39 smithi130 ceph-mon[32704]: pgmap v418: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:35:40.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:39 smithi102 ceph-mon[28094]: pgmap v418: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:35:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:41 smithi112 ceph-mon[32769]: pgmap v419: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:35:42.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:41 smithi130 ceph-mon[32704]: pgmap v419: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:35:42.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:41 smithi102 ceph-mon[28094]: pgmap v419: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:35:44.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:43 smithi112 ceph-mon[32769]: pgmap v420: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:35:44.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:43 smithi130 ceph-mon[32704]: pgmap v420: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:35:44.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:43 smithi102 ceph-mon[28094]: pgmap v420: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:35:46.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:45 smithi112 ceph-mon[32769]: pgmap v421: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:35:46.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:45 smithi130 ceph-mon[32704]: pgmap v421: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:35:46.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:45 smithi102 ceph-mon[28094]: pgmap v421: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:35:47.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:46 smithi112 ceph-mon[32769]: pgmap v422: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:35:47.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:46 smithi130 ceph-mon[32704]: pgmap v422: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:35:47.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:46 smithi102 ceph-mon[28094]: pgmap v422: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:35:50.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:49 smithi112 ceph-mon[32769]: pgmap v423: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:35:50.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:49 smithi130 ceph-mon[32704]: pgmap v423: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:35:50.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:49 smithi102 ceph-mon[28094]: pgmap v423: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:35:52.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:51 smithi112 ceph-mon[32769]: pgmap v424: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:35:52.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:51 smithi130 ceph-mon[32704]: pgmap v424: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:35:52.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:51 smithi102 ceph-mon[28094]: pgmap v424: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:35:54.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:53 smithi112 ceph-mon[32769]: pgmap v425: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:35:54.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:53 smithi130 ceph-mon[32704]: pgmap v425: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:35:54.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:53 smithi102 ceph-mon[28094]: pgmap v425: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:35:55.951 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:55 smithi102 ceph-mon[28094]: pgmap v426: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:35:56.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:55 smithi112 ceph-mon[32769]: pgmap v426: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:35:56.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:55 smithi130 ceph-mon[32704]: pgmap v426: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:35:58.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:57 smithi112 ceph-mon[32769]: pgmap v427: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:35:58.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:57 smithi130 ceph-mon[32704]: pgmap v427: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:35:58.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:57 smithi102 ceph-mon[28094]: pgmap v427: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:36:00.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:35:59 smithi112 ceph-mon[32769]: pgmap v428: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:00.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:35:59 smithi130 ceph-mon[32704]: pgmap v428: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:00.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:35:59 smithi102 ceph-mon[28094]: pgmap v428: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:02.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:01 smithi112 ceph-mon[32769]: pgmap v429: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:02.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:01 smithi130 ceph-mon[32704]: pgmap v429: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:02.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:01 smithi102 ceph-mon[28094]: pgmap v429: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:04.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:03 smithi112 ceph-mon[32769]: pgmap v430: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:04.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:03 smithi130 ceph-mon[32704]: pgmap v430: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:04.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:03 smithi102 ceph-mon[28094]: pgmap v430: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:06.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:05 smithi112 ceph-mon[32769]: pgmap v431: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:06.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:05 smithi130 ceph-mon[32704]: pgmap v431: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:06.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:05 smithi102 ceph-mon[28094]: pgmap v431: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:08.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:07 smithi112 ceph-mon[32769]: pgmap v432: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:08.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:07 smithi130 ceph-mon[32704]: pgmap v432: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:08.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:07 smithi102 ceph-mon[28094]: pgmap v432: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:10.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:09 smithi112 ceph-mon[32769]: pgmap v433: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:36:10.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:09 smithi130 ceph-mon[32704]: pgmap v433: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:36:10.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:09 smithi102 ceph-mon[28094]: pgmap v433: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:36:12.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:11 smithi112 ceph-mon[32769]: pgmap v434: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:36:12.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:11 smithi130 ceph-mon[32704]: pgmap v434: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:36:12.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:11 smithi102 ceph-mon[28094]: pgmap v434: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:36:13.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:12 smithi130 ceph-mon[32704]: pgmap v435: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:36:13.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:12 smithi102 ceph-mon[28094]: pgmap v435: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:36:13.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:12 smithi112 ceph-mon[32769]: pgmap v435: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:36:16.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:15 smithi112 ceph-mon[32769]: pgmap v436: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:36:16.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:15 smithi130 ceph-mon[32704]: pgmap v436: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:36:16.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:15 smithi102 ceph-mon[28094]: pgmap v436: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:36:18.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:17 smithi112 ceph-mon[32769]: pgmap v437: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:36:18.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:17 smithi130 ceph-mon[32704]: pgmap v437: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:36:18.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:17 smithi102 ceph-mon[28094]: pgmap v437: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:36:20.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:19 smithi112 ceph-mon[32769]: pgmap v438: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:36:20.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:19 smithi130 ceph-mon[32704]: pgmap v438: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:36:20.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:19 smithi102 ceph-mon[28094]: pgmap v438: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:36:22.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:21 smithi112 ceph-mon[32769]: pgmap v439: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:22.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:21 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:36:22.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:21 smithi130 ceph-mon[32704]: pgmap v439: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:22.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:21 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:36:22.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:21 smithi102 ceph-mon[28094]: pgmap v439: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:22.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:21 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:36:24.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:23 smithi112 ceph-mon[32769]: pgmap v440: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:24.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:23 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:36:24.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:23 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:36:24.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:23 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:36:24.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:23 smithi130 ceph-mon[32704]: pgmap v440: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:24.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:23 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:36:24.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:23 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:36:24.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:23 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:36:24.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:23 smithi102 ceph-mon[28094]: pgmap v440: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:24.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:23 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:36:24.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:23 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:36:24.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:23 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:36:26.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:25 smithi112 ceph-mon[32769]: pgmap v441: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:26.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:25 smithi130 ceph-mon[32704]: pgmap v441: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:26.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:25 smithi102 ceph-mon[28094]: pgmap v441: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:27 smithi112 ceph-mon[32769]: pgmap v442: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:28.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:27 smithi130 ceph-mon[32704]: pgmap v442: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:28.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:27 smithi102 ceph-mon[28094]: pgmap v442: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:30.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:29 smithi112 ceph-mon[32769]: pgmap v443: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:30.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:29 smithi130 ceph-mon[32704]: pgmap v443: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:29 smithi102 ceph-mon[28094]: pgmap v443: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:31 smithi112 ceph-mon[32769]: pgmap v444: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:32.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:31 smithi130 ceph-mon[32704]: pgmap v444: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:32.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:31 smithi102 ceph-mon[28094]: pgmap v444: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:34.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:33 smithi112 ceph-mon[32769]: pgmap v445: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:34.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:33 smithi130 ceph-mon[32704]: pgmap v445: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:34.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:33 smithi102 ceph-mon[28094]: pgmap v445: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:34 smithi130 ceph-mon[32704]: pgmap v446: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:35.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:34 smithi102 ceph-mon[28094]: pgmap v446: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:35.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:34 smithi112 ceph-mon[32769]: pgmap v446: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:38.031 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:37 smithi112 ceph-mon[32769]: pgmap v447: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:38.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:37 smithi130 ceph-mon[32704]: pgmap v447: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:38.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:37 smithi102 ceph-mon[28094]: pgmap v447: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:40.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:39 smithi112 ceph-mon[32769]: pgmap v448: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:40.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:39 smithi130 ceph-mon[32704]: pgmap v448: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:40.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:39 smithi102 ceph-mon[28094]: pgmap v448: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:41 smithi112 ceph-mon[32769]: pgmap v449: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:42.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:41 smithi130 ceph-mon[32704]: pgmap v449: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:42.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:41 smithi102 ceph-mon[28094]: pgmap v449: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:44.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:43 smithi112 ceph-mon[32769]: pgmap v450: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:44.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:43 smithi130 ceph-mon[32704]: pgmap v450: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:44.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:43 smithi102 ceph-mon[28094]: pgmap v450: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:46.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:45 smithi112 ceph-mon[32769]: pgmap v451: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:46.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:45 smithi130 ceph-mon[32704]: pgmap v451: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:46.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:45 smithi102 ceph-mon[28094]: pgmap v451: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:48.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:47 smithi112 ceph-mon[32769]: pgmap v452: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:48.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:47 smithi130 ceph-mon[32704]: pgmap v452: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:48.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:47 smithi102 ceph-mon[28094]: pgmap v452: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:50.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:49 smithi112 ceph-mon[32769]: pgmap v453: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:50.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:49 smithi130 ceph-mon[32704]: pgmap v453: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:50.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:49 smithi102 ceph-mon[28094]: pgmap v453: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:36:52.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:51 smithi112 ceph-mon[32769]: pgmap v454: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:52.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:51 smithi130 ceph-mon[32704]: pgmap v454: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:52.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:51 smithi102 ceph-mon[28094]: pgmap v454: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:36:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:53 smithi112 ceph-mon[32769]: pgmap v455: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:54.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:53 smithi130 ceph-mon[32704]: pgmap v455: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:54.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:53 smithi102 ceph-mon[28094]: pgmap v455: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:36:55.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:54 smithi112 ceph-mon[32769]: pgmap v456: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:55.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:54 smithi102 ceph-mon[28094]: pgmap v456: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:55.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:54 smithi130 ceph-mon[32704]: pgmap v456: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:36:58.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:57 smithi112 ceph-mon[32769]: pgmap v457: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:58.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:57 smithi130 ceph-mon[32704]: pgmap v457: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:36:58.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:57 smithi102 ceph-mon[28094]: pgmap v457: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:00.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:36:59 smithi112 ceph-mon[32769]: pgmap v458: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:00.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:36:59 smithi130 ceph-mon[32704]: pgmap v458: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:00.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:36:59 smithi102 ceph-mon[28094]: pgmap v458: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:02.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:01 smithi112 ceph-mon[32769]: pgmap v459: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:02.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:01 smithi130 ceph-mon[32704]: pgmap v459: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:02.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:01 smithi102 ceph-mon[28094]: pgmap v459: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:04.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:03 smithi112 ceph-mon[32769]: pgmap v460: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:04.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:03 smithi130 ceph-mon[32704]: pgmap v460: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:04.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:03 smithi102 ceph-mon[28094]: pgmap v460: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:06.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:05 smithi112 ceph-mon[32769]: pgmap v461: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:06.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:05 smithi130 ceph-mon[32704]: pgmap v461: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:06.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:05 smithi102 ceph-mon[28094]: pgmap v461: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:08.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:07 smithi112 ceph-mon[32769]: pgmap v462: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:08.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:07 smithi130 ceph-mon[32704]: pgmap v462: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:08.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:07 smithi102 ceph-mon[28094]: pgmap v462: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:10.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:09 smithi112 ceph-mon[32769]: pgmap v463: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:10.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:09 smithi130 ceph-mon[32704]: pgmap v463: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:10.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:09 smithi102 ceph-mon[28094]: pgmap v463: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:12.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:11 smithi112 ceph-mon[32769]: pgmap v464: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:12.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:11 smithi130 ceph-mon[32704]: pgmap v464: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:12.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:11 smithi102 ceph-mon[28094]: pgmap v464: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:13.023 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:12 smithi130 ceph-mon[32704]: pgmap v465: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:13.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:12 smithi112 ceph-mon[32769]: pgmap v465: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:13.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:12 smithi102 ceph-mon[28094]: pgmap v465: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:16.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:15 smithi112 ceph-mon[32769]: pgmap v466: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:16.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:15 smithi130 ceph-mon[32704]: pgmap v466: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:16.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:15 smithi102 ceph-mon[28094]: pgmap v466: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:18.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:17 smithi112 ceph-mon[32769]: pgmap v467: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:18.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:17 smithi130 ceph-mon[32704]: pgmap v467: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:18.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:17 smithi102 ceph-mon[28094]: pgmap v467: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:20.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:19 smithi112 ceph-mon[32769]: pgmap v468: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:20.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:19 smithi130 ceph-mon[32704]: pgmap v468: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:20.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:19 smithi102 ceph-mon[28094]: pgmap v468: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:22.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:21 smithi112 ceph-mon[32769]: pgmap v469: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:22.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:21 smithi130 ceph-mon[32704]: pgmap v469: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:22.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:21 smithi102 ceph-mon[28094]: pgmap v469: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:24.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:23 smithi112 ceph-mon[32769]: pgmap v470: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:24.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:23 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:37:24.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:23 smithi130 ceph-mon[32704]: pgmap v470: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:24.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:23 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:37:24.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:23 smithi102 ceph-mon[28094]: pgmap v470: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:24.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:23 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:37:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:37:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:37:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:37:25.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:37:25.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:37:25.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:37:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:37:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:37:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:37:26.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:25 smithi112 ceph-mon[32769]: pgmap v471: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:26.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:25 smithi130 ceph-mon[32704]: pgmap v471: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:26.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:25 smithi102 ceph-mon[28094]: pgmap v471: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:28.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:27 smithi112 ceph-mon[32769]: pgmap v472: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:28.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:27 smithi130 ceph-mon[32704]: pgmap v472: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:28.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:27 smithi102 ceph-mon[28094]: pgmap v472: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:29.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:28 smithi112 ceph-mon[32769]: pgmap v473: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:29.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:28 smithi130 ceph-mon[32704]: pgmap v473: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:29.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:28 smithi102 ceph-mon[28094]: pgmap v473: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:32.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:31 smithi112 ceph-mon[32769]: pgmap v474: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:32.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:31 smithi130 ceph-mon[32704]: pgmap v474: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:32.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:31 smithi102 ceph-mon[28094]: pgmap v474: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:34.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:33 smithi112 ceph-mon[32769]: pgmap v475: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:34.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:33 smithi130 ceph-mon[32704]: pgmap v475: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:34.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:33 smithi102 ceph-mon[28094]: pgmap v475: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:36.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:35 smithi112 ceph-mon[32769]: pgmap v476: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:36.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:35 smithi130 ceph-mon[32704]: pgmap v476: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:36.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:35 smithi102 ceph-mon[28094]: pgmap v476: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:38.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:37 smithi112 ceph-mon[32769]: pgmap v477: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:38.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:37 smithi130 ceph-mon[32704]: pgmap v477: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:38.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:37 smithi102 ceph-mon[28094]: pgmap v477: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:40.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:39 smithi112 ceph-mon[32769]: pgmap v478: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:40.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:39 smithi130 ceph-mon[32704]: pgmap v478: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:40.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:39 smithi102 ceph-mon[28094]: pgmap v478: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:41 smithi112 ceph-mon[32769]: pgmap v479: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:42.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:41 smithi130 ceph-mon[32704]: pgmap v479: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:42.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:41 smithi102 ceph-mon[28094]: pgmap v479: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:43.023 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:42 smithi130 ceph-mon[32704]: pgmap v480: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:43.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:42 smithi112 ceph-mon[32769]: pgmap v480: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:43.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:42 smithi102 ceph-mon[28094]: pgmap v480: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:46.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:45 smithi112 ceph-mon[32769]: pgmap v481: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:46.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:45 smithi130 ceph-mon[32704]: pgmap v481: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:46.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:45 smithi102 ceph-mon[28094]: pgmap v481: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:48.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:47 smithi112 ceph-mon[32769]: pgmap v482: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:48.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:47 smithi130 ceph-mon[32704]: pgmap v482: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:48.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:47 smithi102 ceph-mon[28094]: pgmap v482: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:50.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:49 smithi112 ceph-mon[32769]: pgmap v483: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:50.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:49 smithi130 ceph-mon[32704]: pgmap v483: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:50.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:49 smithi102 ceph-mon[28094]: pgmap v483: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:37:52.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:51 smithi112 ceph-mon[32769]: pgmap v484: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:52.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:51 smithi130 ceph-mon[32704]: pgmap v484: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:52.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:51 smithi102 ceph-mon[28094]: pgmap v484: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:37:54.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:53 smithi112 ceph-mon[32769]: pgmap v485: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:54.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:53 smithi130 ceph-mon[32704]: pgmap v485: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:54.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:53 smithi102 ceph-mon[28094]: pgmap v485: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:37:55.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:54 smithi112 ceph-mon[32769]: pgmap v486: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:55.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:54 smithi102 ceph-mon[28094]: pgmap v486: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:55.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:54 smithi130 ceph-mon[32704]: pgmap v486: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:37:58.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:57 smithi112 ceph-mon[32769]: pgmap v487: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:58.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:57 smithi130 ceph-mon[32704]: pgmap v487: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:37:58.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:57 smithi102 ceph-mon[28094]: pgmap v487: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:00.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:37:59 smithi112 ceph-mon[32769]: pgmap v488: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:00.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:37:59 smithi130 ceph-mon[32704]: pgmap v488: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:00.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:37:59 smithi102 ceph-mon[28094]: pgmap v488: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:02.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:01 smithi112 ceph-mon[32769]: pgmap v489: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:02.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:01 smithi130 ceph-mon[32704]: pgmap v489: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:02.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:01 smithi102 ceph-mon[28094]: pgmap v489: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:04.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:03 smithi112 ceph-mon[32769]: pgmap v490: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:04.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:03 smithi130 ceph-mon[32704]: pgmap v490: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:04.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:03 smithi102 ceph-mon[28094]: pgmap v490: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:05.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:04 smithi112 ceph-mon[32769]: pgmap v491: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:05.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:04 smithi130 ceph-mon[32704]: pgmap v491: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:05.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:04 smithi102 ceph-mon[28094]: pgmap v491: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:08.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:07 smithi112 ceph-mon[32769]: pgmap v492: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:08.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:07 smithi130 ceph-mon[32704]: pgmap v492: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:08.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:07 smithi102 ceph-mon[28094]: pgmap v492: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:10.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:09 smithi112 ceph-mon[32769]: pgmap v493: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:10.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:09 smithi130 ceph-mon[32704]: pgmap v493: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:10.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:09 smithi102 ceph-mon[28094]: pgmap v493: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:12.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:11 smithi112 ceph-mon[32769]: pgmap v494: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:12.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:11 smithi130 ceph-mon[32704]: pgmap v494: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:12.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:11 smithi102 ceph-mon[28094]: pgmap v494: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:13.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:12 smithi130 ceph-mon[32704]: pgmap v495: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:13.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:12 smithi102 ceph-mon[28094]: pgmap v495: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:13.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:12 smithi112 ceph-mon[32769]: pgmap v495: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:16.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:15 smithi112 ceph-mon[32769]: pgmap v496: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:16.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:15 smithi130 ceph-mon[32704]: pgmap v496: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:16.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:15 smithi102 ceph-mon[28094]: pgmap v496: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:18.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:17 smithi112 ceph-mon[32769]: pgmap v497: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:18.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:17 smithi130 ceph-mon[32704]: pgmap v497: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:18.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:17 smithi102 ceph-mon[28094]: pgmap v497: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:38:18.359+0000 7f8060c00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f80400034c0 con 0x7f80641028c0 2024-04-26T20:38:18.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:38:18.360+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f80400034c0 con 0x7f80641028c0 2024-04-26T20:38:19.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:18 smithi130 ceph-mon[32704]: pgmap v498: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:19.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:18 smithi102 ceph-mon[28094]: pgmap v498: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:19.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:18 smithi112 ceph-mon[32769]: pgmap v498: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:22.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:21 smithi112 ceph-mon[32769]: pgmap v499: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:22.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:21 smithi130 ceph-mon[32704]: pgmap v499: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:22.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:21 smithi102 ceph-mon[28094]: pgmap v499: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:24.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:23 smithi112 ceph-mon[32769]: pgmap v500: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:24.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:23 smithi130 ceph-mon[32704]: pgmap v500: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:24.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:23 smithi102 ceph-mon[28094]: pgmap v500: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:25.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:24 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:38:25.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:24 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:38:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:24 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:38:26.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:25 smithi112 ceph-mon[32769]: pgmap v501: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:26.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:25 smithi130 ceph-mon[32704]: pgmap v501: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:26.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:25 smithi102 ceph-mon[28094]: pgmap v501: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:27.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:38:27.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:38:27.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:38:27.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:26 smithi112 ceph-mon[32769]: pgmap v502: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:27.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:38:27.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:38:27.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:38:27.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:26 smithi130 ceph-mon[32704]: pgmap v502: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:27.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:38:27.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:38:27.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:38:27.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:26 smithi102 ceph-mon[28094]: pgmap v502: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:30.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:29 smithi112 ceph-mon[32769]: pgmap v503: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:30.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:29 smithi130 ceph-mon[32704]: pgmap v503: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:29 smithi102 ceph-mon[28094]: pgmap v503: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:30 smithi130 ceph-mon[32704]: pgmap v504: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:31.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:30 smithi102 ceph-mon[28094]: pgmap v504: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:31.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:30 smithi112 ceph-mon[32769]: pgmap v504: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:34.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:33 smithi112 ceph-mon[32769]: pgmap v505: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:34.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:33 smithi130 ceph-mon[32704]: pgmap v505: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:34.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:33 smithi102 ceph-mon[28094]: pgmap v505: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:34 smithi130 ceph-mon[32704]: pgmap v506: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:35.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:34 smithi102 ceph-mon[28094]: pgmap v506: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:35.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:34 smithi112 ceph-mon[32769]: pgmap v506: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:38.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:37 smithi112 ceph-mon[32769]: pgmap v507: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:38.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:37 smithi130 ceph-mon[32704]: pgmap v507: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:38.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:37 smithi102 ceph-mon[28094]: pgmap v507: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:39.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:38 smithi130 ceph-mon[32704]: pgmap v508: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:39.252 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:38 smithi102 ceph-mon[28094]: pgmap v508: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:39.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:38 smithi112 ceph-mon[32769]: pgmap v508: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:42.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:41 smithi112 ceph-mon[32769]: pgmap v509: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:42.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:41 smithi130 ceph-mon[32704]: pgmap v509: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:42.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:41 smithi102 ceph-mon[28094]: pgmap v509: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:43.023 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:42 smithi130 ceph-mon[32704]: pgmap v510: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:43.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:42 smithi112 ceph-mon[32769]: pgmap v510: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:43.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:42 smithi102 ceph-mon[28094]: pgmap v510: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:45.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:44 smithi130 ceph-mon[32704]: pgmap v511: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:45.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:44 smithi102 ceph-mon[28094]: pgmap v511: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:45.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:44 smithi112 ceph-mon[32769]: pgmap v511: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:47.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:46 smithi130 ceph-mon[32704]: pgmap v512: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:47.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:46 smithi102 ceph-mon[28094]: pgmap v512: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:47.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:46 smithi112 ceph-mon[32769]: pgmap v512: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:49.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:48 smithi130 ceph-mon[32704]: pgmap v513: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:49.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:48 smithi102 ceph-mon[28094]: pgmap v513: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:49.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:48 smithi112 ceph-mon[32769]: pgmap v513: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:51.079 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:50 smithi112 ceph-mon[32769]: pgmap v514: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:51.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:50 smithi130 ceph-mon[32704]: pgmap v514: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:51.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:50 smithi102 ceph-mon[28094]: pgmap v514: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:38:53.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:52 smithi130 ceph-mon[32704]: pgmap v515: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:53.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:52 smithi102 ceph-mon[28094]: pgmap v515: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:53.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:52 smithi112 ceph-mon[32769]: pgmap v515: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:38:55.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:54 smithi130 ceph-mon[32704]: pgmap v516: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:55.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:54 smithi102 ceph-mon[28094]: pgmap v516: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:55.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:54 smithi112 ceph-mon[32769]: pgmap v516: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:38:57.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:56 smithi130 ceph-mon[32704]: pgmap v517: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:57.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:56 smithi102 ceph-mon[28094]: pgmap v517: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:57.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:56 smithi112 ceph-mon[32769]: pgmap v517: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:38:59.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:38:58 smithi130 ceph-mon[32704]: pgmap v518: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:59.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:38:58 smithi102 ceph-mon[28094]: pgmap v518: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:38:59.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:38:58 smithi112 ceph-mon[32769]: pgmap v518: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:01.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:00 smithi130 ceph-mon[32704]: pgmap v519: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:01.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:00 smithi102 ceph-mon[28094]: pgmap v519: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:01.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:00 smithi112 ceph-mon[32769]: pgmap v519: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:03.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:02 smithi130 ceph-mon[32704]: pgmap v520: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:03.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:02 smithi102 ceph-mon[28094]: pgmap v520: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:03.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:02 smithi112 ceph-mon[32769]: pgmap v520: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:05.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:04 smithi130 ceph-mon[32704]: pgmap v521: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:39:05.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:04 smithi102 ceph-mon[28094]: pgmap v521: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:39:05.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:04 smithi112 ceph-mon[32769]: pgmap v521: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:39:07.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:06 smithi130 ceph-mon[32704]: pgmap v522: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-26T20:39:07.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:06 smithi102 ceph-mon[28094]: pgmap v522: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-26T20:39:07.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:06 smithi112 ceph-mon[32769]: pgmap v522: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-26T20:39:09.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:08 smithi130 ceph-mon[32704]: pgmap v523: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:39:09.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:08 smithi102 ceph-mon[28094]: pgmap v523: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:39:09.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:08 smithi112 ceph-mon[32769]: pgmap v523: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:39:11.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:10 smithi130 ceph-mon[32704]: pgmap v524: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-26T20:39:11.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:10 smithi102 ceph-mon[28094]: pgmap v524: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-26T20:39:11.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:10 smithi112 ceph-mon[32769]: pgmap v524: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-26T20:39:13.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:12 smithi130 ceph-mon[32704]: pgmap v525: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-04-26T20:39:13.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:12 smithi102 ceph-mon[28094]: pgmap v525: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-04-26T20:39:13.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:12 smithi112 ceph-mon[32769]: pgmap v525: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-04-26T20:39:15.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:14 smithi130 ceph-mon[32704]: pgmap v526: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:39:15.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:14 smithi102 ceph-mon[28094]: pgmap v526: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:39:15.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:14 smithi112 ceph-mon[32769]: pgmap v526: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:39:17.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:16 smithi130 ceph-mon[32704]: pgmap v527: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:17.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:16 smithi102 ceph-mon[28094]: pgmap v527: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:17.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:16 smithi112 ceph-mon[32769]: pgmap v527: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:19.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:18 smithi130 ceph-mon[32704]: pgmap v528: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:19.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:18 smithi102 ceph-mon[28094]: pgmap v528: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:19.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:18 smithi112 ceph-mon[32769]: pgmap v528: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:21.079 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:20 smithi112 ceph-mon[32769]: pgmap v529: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:21.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:20 smithi130 ceph-mon[32704]: pgmap v529: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:21.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:20 smithi102 ceph-mon[28094]: pgmap v529: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:23.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:23 smithi112 ceph-mon[32769]: pgmap v530: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:23.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:23 smithi130 ceph-mon[32704]: pgmap v530: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:23.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:23 smithi102 ceph-mon[28094]: pgmap v530: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:25.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:24 smithi130 ceph-mon[32704]: pgmap v531: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:25.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:24 smithi102 ceph-mon[28094]: pgmap v531: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:25.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:24 smithi112 ceph-mon[32769]: pgmap v531: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:26.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:26 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:39:26.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:26 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:39:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:26 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:39:27.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:27 smithi112 ceph-mon[32769]: pgmap v532: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:27.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:27 smithi130 ceph-mon[32704]: pgmap v532: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:27.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:27 smithi102 ceph-mon[28094]: pgmap v532: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:28.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:28 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:39:28.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:28 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:39:28.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:28 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:39:28.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:39:28.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:39:28.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:28 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:39:28.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:39:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:39:28.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:28 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:39:29.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:29 smithi112 ceph-mon[32769]: pgmap v533: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:29.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:29 smithi130 ceph-mon[32704]: pgmap v533: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:29.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:29 smithi102 ceph-mon[28094]: pgmap v533: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:31.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:30 smithi130 ceph-mon[32704]: pgmap v534: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:31.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:30 smithi102 ceph-mon[28094]: pgmap v534: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:31.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:30 smithi112 ceph-mon[32769]: pgmap v534: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:33.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:32 smithi130 ceph-mon[32704]: pgmap v535: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:32 smithi102 ceph-mon[28094]: pgmap v535: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:33.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:32 smithi112 ceph-mon[32769]: pgmap v535: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:34 smithi130 ceph-mon[32704]: pgmap v536: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:39:35.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:34 smithi102 ceph-mon[28094]: pgmap v536: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:39:35.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:34 smithi112 ceph-mon[32769]: pgmap v536: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:39:37.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:36 smithi130 ceph-mon[32704]: pgmap v537: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:37.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:36 smithi102 ceph-mon[28094]: pgmap v537: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:37.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:36 smithi112 ceph-mon[32769]: pgmap v537: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:39.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:38 smithi130 ceph-mon[32704]: pgmap v538: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:39.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:38 smithi102 ceph-mon[28094]: pgmap v538: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:39.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:38 smithi112 ceph-mon[32769]: pgmap v538: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:41.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:40 smithi130 ceph-mon[32704]: pgmap v539: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:41.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:40 smithi102 ceph-mon[28094]: pgmap v539: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:41.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:40 smithi112 ceph-mon[32769]: pgmap v539: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:43.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:42 smithi130 ceph-mon[32704]: pgmap v540: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:43.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:42 smithi102 ceph-mon[28094]: pgmap v540: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:43.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:42 smithi112 ceph-mon[32769]: pgmap v540: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:45.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:44 smithi130 ceph-mon[32704]: pgmap v541: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:39:45.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:44 smithi102 ceph-mon[28094]: pgmap v541: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:39:45.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:44 smithi112 ceph-mon[32769]: pgmap v541: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:39:47.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:46 smithi130 ceph-mon[32704]: pgmap v542: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:47.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:46 smithi102 ceph-mon[28094]: pgmap v542: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:47.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:46 smithi112 ceph-mon[32769]: pgmap v542: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:49.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:48 smithi130 ceph-mon[32704]: pgmap v543: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:49.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:48 smithi102 ceph-mon[28094]: pgmap v543: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:49.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:48 smithi112 ceph-mon[32769]: pgmap v543: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:51.079 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:50 smithi112 ceph-mon[32769]: pgmap v544: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:51.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:50 smithi130 ceph-mon[32704]: pgmap v544: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:51.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:50 smithi102 ceph-mon[28094]: pgmap v544: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:39:53.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:52 smithi130 ceph-mon[32704]: pgmap v545: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:53.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:52 smithi102 ceph-mon[28094]: pgmap v545: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:53.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:52 smithi112 ceph-mon[32769]: pgmap v545: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:39:55.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:54 smithi130 ceph-mon[32704]: pgmap v546: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:39:55.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:54 smithi102 ceph-mon[28094]: pgmap v546: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:39:55.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:54 smithi112 ceph-mon[32769]: pgmap v546: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:39:57.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:56 smithi130 ceph-mon[32704]: pgmap v547: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:57.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:56 smithi102 ceph-mon[28094]: pgmap v547: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:57.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:56 smithi112 ceph-mon[32769]: pgmap v547: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:39:59.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:39:58 smithi130 ceph-mon[32704]: pgmap v548: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:59.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:39:58 smithi102 ceph-mon[28094]: pgmap v548: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:39:59.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:39:58 smithi112 ceph-mon[32769]: pgmap v548: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:00.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:00 smithi130 ceph-mon[32704]: overall HEALTH_OK 2024-04-26T20:40:00.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:00 smithi102 ceph-mon[28094]: overall HEALTH_OK 2024-04-26T20:40:00.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:00 smithi112 ceph-mon[32769]: overall HEALTH_OK 2024-04-26T20:40:01.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:01 smithi130 ceph-mon[32704]: pgmap v549: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:01.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:01 smithi102 ceph-mon[28094]: pgmap v549: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:01.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:01 smithi112 ceph-mon[32769]: pgmap v549: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:03.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:02 smithi130 ceph-mon[32704]: pgmap v550: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:03.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:02 smithi102 ceph-mon[28094]: pgmap v550: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:03.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:02 smithi112 ceph-mon[32769]: pgmap v550: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:05.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:04 smithi130 ceph-mon[32704]: pgmap v551: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:05.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:04 smithi102 ceph-mon[28094]: pgmap v551: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:05.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:04 smithi112 ceph-mon[32769]: pgmap v551: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:07.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:06 smithi130 ceph-mon[32704]: pgmap v552: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:07.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:06 smithi102 ceph-mon[28094]: pgmap v552: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:07.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:06 smithi112 ceph-mon[32769]: pgmap v552: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:09.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:08 smithi112 ceph-mon[32769]: pgmap v553: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:09.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:08 smithi130 ceph-mon[32704]: pgmap v553: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:09.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:08 smithi102 ceph-mon[28094]: pgmap v553: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:11.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:10 smithi130 ceph-mon[32704]: pgmap v554: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:11.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:10 smithi102 ceph-mon[28094]: pgmap v554: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:11.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:10 smithi112 ceph-mon[32769]: pgmap v554: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:13.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:12 smithi130 ceph-mon[32704]: pgmap v555: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:13.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:12 smithi102 ceph-mon[28094]: pgmap v555: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:13.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:12 smithi112 ceph-mon[32769]: pgmap v555: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:15.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:14 smithi130 ceph-mon[32704]: pgmap v556: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:15.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:14 smithi102 ceph-mon[28094]: pgmap v556: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:15.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:14 smithi112 ceph-mon[32769]: pgmap v556: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:17.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:16 smithi130 ceph-mon[32704]: pgmap v557: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:17.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:16 smithi102 ceph-mon[28094]: pgmap v557: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:17.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:16 smithi112 ceph-mon[32769]: pgmap v557: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:19.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:18 smithi130 ceph-mon[32704]: pgmap v558: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:19.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:18 smithi102 ceph-mon[28094]: pgmap v558: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:19.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:18 smithi112 ceph-mon[32769]: pgmap v558: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:21.079 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:20 smithi112 ceph-mon[32769]: pgmap v559: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:21.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:20 smithi130 ceph-mon[32704]: pgmap v559: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:21.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:20 smithi102 ceph-mon[28094]: pgmap v559: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:23.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:22 smithi130 ceph-mon[32704]: pgmap v560: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:23.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:22 smithi102 ceph-mon[28094]: pgmap v560: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:23.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:22 smithi112 ceph-mon[32769]: pgmap v560: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:25.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:24 smithi130 ceph-mon[32704]: pgmap v561: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:25.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:24 smithi102 ceph-mon[28094]: pgmap v561: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:25.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:24 smithi112 ceph-mon[32769]: pgmap v561: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:27.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:26 smithi130 ceph-mon[32704]: pgmap v562: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:27.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:26 smithi102 ceph-mon[28094]: pgmap v562: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:27.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:26 smithi112 ceph-mon[32769]: pgmap v562: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:28.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:27 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:40:28.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:27 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:40:28.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:27 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:40:29.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:28 smithi130 ceph-mon[32704]: pgmap v563: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:29.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:28 smithi102 ceph-mon[28094]: pgmap v563: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:29.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:28 smithi112 ceph-mon[32769]: pgmap v563: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:30.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:40:30.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:40:30.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:40:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:40:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:40:30.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:40:30.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:40:30.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:40:30.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:40:31.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:30 smithi130 ceph-mon[32704]: pgmap v564: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:31.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:30 smithi102 ceph-mon[28094]: pgmap v564: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:31.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:30 smithi112 ceph-mon[32769]: pgmap v564: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:32 smithi130 ceph-mon[32704]: pgmap v565: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:33.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:32 smithi102 ceph-mon[28094]: pgmap v565: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:33.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:32 smithi112 ceph-mon[32769]: pgmap v565: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:34 smithi130 ceph-mon[32704]: pgmap v566: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:35.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:34 smithi102 ceph-mon[28094]: pgmap v566: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:35.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:34 smithi112 ceph-mon[32769]: pgmap v566: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:37.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:36 smithi130 ceph-mon[32704]: pgmap v567: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:37.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:36 smithi102 ceph-mon[28094]: pgmap v567: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:37.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:36 smithi112 ceph-mon[32769]: pgmap v567: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:39.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:38 smithi130 ceph-mon[32704]: pgmap v568: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:39.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:38 smithi102 ceph-mon[28094]: pgmap v568: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:39.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:38 smithi112 ceph-mon[32769]: pgmap v568: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:41.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:40 smithi130 ceph-mon[32704]: pgmap v569: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:41.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:40 smithi102 ceph-mon[28094]: pgmap v569: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:41.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:40 smithi112 ceph-mon[32769]: pgmap v569: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:43.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:42 smithi130 ceph-mon[32704]: pgmap v570: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:43.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:42 smithi102 ceph-mon[28094]: pgmap v570: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:43.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:42 smithi112 ceph-mon[32769]: pgmap v570: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:45.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:44 smithi130 ceph-mon[32704]: pgmap v571: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:45.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:44 smithi102 ceph-mon[28094]: pgmap v571: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:45.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:44 smithi112 ceph-mon[32769]: pgmap v571: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:47.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:46 smithi130 ceph-mon[32704]: pgmap v572: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:47.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:46 smithi102 ceph-mon[28094]: pgmap v572: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:47.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:46 smithi112 ceph-mon[32769]: pgmap v572: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:49.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:48 smithi130 ceph-mon[32704]: pgmap v573: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:49.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:48 smithi102 ceph-mon[28094]: pgmap v573: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:49.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:48 smithi112 ceph-mon[32769]: pgmap v573: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:51.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:50 smithi130 ceph-mon[32704]: pgmap v574: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:51.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:50 smithi102 ceph-mon[28094]: pgmap v574: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:51.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:50 smithi112 ceph-mon[32769]: pgmap v574: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:40:53.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:52 smithi130 ceph-mon[32704]: pgmap v575: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:53.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:52 smithi102 ceph-mon[28094]: pgmap v575: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:53.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:52 smithi112 ceph-mon[32769]: pgmap v575: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:40:55.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:54 smithi130 ceph-mon[32704]: pgmap v576: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:55.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:54 smithi102 ceph-mon[28094]: pgmap v576: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:55.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:54 smithi112 ceph-mon[32769]: pgmap v576: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:40:57.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:56 smithi130 ceph-mon[32704]: pgmap v577: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:57.292 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:56 smithi112 ceph-mon[32769]: pgmap v577: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:57.292 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:56 smithi102 ceph-mon[28094]: pgmap v577: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:40:59.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:40:58 smithi130 ceph-mon[32704]: pgmap v578: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:59.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:40:58 smithi102 ceph-mon[28094]: pgmap v578: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:40:59.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:40:58 smithi112 ceph-mon[32769]: pgmap v578: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:01.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:00 smithi130 ceph-mon[32704]: pgmap v579: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:01.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:00 smithi102 ceph-mon[28094]: pgmap v579: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:01.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:00 smithi112 ceph-mon[32769]: pgmap v579: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:03.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:02 smithi130 ceph-mon[32704]: pgmap v580: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:03.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:02 smithi102 ceph-mon[28094]: pgmap v580: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:03.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:02 smithi112 ceph-mon[32769]: pgmap v580: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:05.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:04 smithi130 ceph-mon[32704]: pgmap v581: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:05.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:04 smithi102 ceph-mon[28094]: pgmap v581: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:05.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:04 smithi112 ceph-mon[32769]: pgmap v581: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:07.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:06 smithi130 ceph-mon[32704]: pgmap v582: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:07.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:06 smithi102 ceph-mon[28094]: pgmap v582: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:07.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:06 smithi112 ceph-mon[32769]: pgmap v582: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:09.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:08 smithi130 ceph-mon[32704]: pgmap v583: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:09.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:08 smithi102 ceph-mon[28094]: pgmap v583: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:09.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:08 smithi112 ceph-mon[32769]: pgmap v583: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:11.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:10 smithi130 ceph-mon[32704]: pgmap v584: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:11.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:10 smithi102 ceph-mon[28094]: pgmap v584: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:11.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:10 smithi112 ceph-mon[32769]: pgmap v584: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:13.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:12 smithi130 ceph-mon[32704]: pgmap v585: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:13.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:12 smithi102 ceph-mon[28094]: pgmap v585: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:13.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:12 smithi112 ceph-mon[32769]: pgmap v585: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:15.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:14 smithi130 ceph-mon[32704]: pgmap v586: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:15.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:14 smithi102 ceph-mon[28094]: pgmap v586: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:15.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:14 smithi112 ceph-mon[32769]: pgmap v586: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:17.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:16 smithi130 ceph-mon[32704]: pgmap v587: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:17.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:16 smithi102 ceph-mon[28094]: pgmap v587: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:17.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:16 smithi112 ceph-mon[32769]: pgmap v587: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:18.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:41:18.389+0000 7f8060c00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f8040005850 con 0x7f80641028c0 2024-04-26T20:41:18.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:41:18.390+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f8040005850 con 0x7f80641028c0 2024-04-26T20:41:19.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:18 smithi130 ceph-mon[32704]: pgmap v588: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:19.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:18 smithi102 ceph-mon[28094]: pgmap v588: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:19.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:18 smithi112 ceph-mon[32769]: pgmap v588: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:21.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:20 smithi130 ceph-mon[32704]: pgmap v589: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:21.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:20 smithi102 ceph-mon[28094]: pgmap v589: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:21.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:20 smithi112 ceph-mon[32769]: pgmap v589: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:23.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:22 smithi130 ceph-mon[32704]: pgmap v590: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:23.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:22 smithi102 ceph-mon[28094]: pgmap v590: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:23.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:22 smithi112 ceph-mon[32769]: pgmap v590: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:24 smithi130 ceph-mon[32704]: pgmap v591: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:25.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:24 smithi102 ceph-mon[28094]: pgmap v591: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:25.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:24 smithi112 ceph-mon[32769]: pgmap v591: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:27.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:26 smithi130 ceph-mon[32704]: pgmap v592: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:27.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:26 smithi102 ceph-mon[28094]: pgmap v592: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:27.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:26 smithi112 ceph-mon[32769]: pgmap v592: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:29.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:28 smithi130 ceph-mon[32704]: pgmap v593: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:29.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:28 smithi102 ceph-mon[28094]: pgmap v593: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:29.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:28 smithi112 ceph-mon[32769]: pgmap v593: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:30.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:29 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:41:30.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:29 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:41:30.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:29 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:41:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-04-26T20:41:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: Unable to set osd_memory_target on smithi130 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-26T20:41:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: Unable to set osd_memory_target on smithi112 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: Adjusting osd_memory_target on smithi102 to 176.4M 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: Unable to set osd_memory_target on smithi102 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:41:31.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:30 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:41:31.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: Unable to set osd_memory_target on smithi130 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: Unable to set osd_memory_target on smithi112 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: Adjusting osd_memory_target on smithi102 to 176.4M 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: Unable to set osd_memory_target on smithi102 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:41:31.150 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:41:31.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:30 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: Unable to set osd_memory_target on smithi130 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: Unable to set osd_memory_target on smithi112 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T20:41:31.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: Adjusting osd_memory_target on smithi102 to 176.4M 2024-04-26T20:41:31.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: Unable to set osd_memory_target on smithi102 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-26T20:41:31.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:41:31.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:41:31.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:30 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:41:32.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:32 smithi112 ceph-mon[32769]: pgmap v594: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:32.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:32 smithi130 ceph-mon[32704]: pgmap v594: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:32.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:32 smithi102 ceph-mon[28094]: pgmap v594: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:34.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:34 smithi112 ceph-mon[32769]: pgmap v595: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:34.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:34 smithi130 ceph-mon[32704]: pgmap v595: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:34.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:34 smithi102 ceph-mon[28094]: pgmap v595: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:36.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:36 smithi112 ceph-mon[32769]: pgmap v596: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:36.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:36 smithi130 ceph-mon[32704]: pgmap v596: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:36.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:36 smithi102 ceph-mon[28094]: pgmap v596: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:38.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:38 smithi112 ceph-mon[32769]: pgmap v597: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:38.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:38 smithi130 ceph-mon[32704]: pgmap v597: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:38.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:38 smithi102 ceph-mon[28094]: pgmap v597: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:40.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:40 smithi112 ceph-mon[32769]: pgmap v598: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:40.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:40 smithi130 ceph-mon[32704]: pgmap v598: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:40.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:40 smithi102 ceph-mon[28094]: pgmap v598: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:42.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:42 smithi112 ceph-mon[32769]: pgmap v599: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:42.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:42 smithi130 ceph-mon[32704]: pgmap v599: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:42.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:42 smithi102 ceph-mon[28094]: pgmap v599: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:44.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:44 smithi112 ceph-mon[32769]: pgmap v600: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:44.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:44 smithi130 ceph-mon[32704]: pgmap v600: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:44.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:44 smithi102 ceph-mon[28094]: pgmap v600: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:46.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:46 smithi130 ceph-mon[32704]: pgmap v601: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:46.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:46 smithi102 ceph-mon[28094]: pgmap v601: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:47.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:46 smithi112 ceph-mon[32769]: pgmap v601: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:48.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:48 smithi130 ceph-mon[32704]: pgmap v602: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:48.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:48 smithi102 ceph-mon[28094]: pgmap v602: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:49.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:48 smithi112 ceph-mon[32769]: pgmap v602: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:50.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:50 smithi130 ceph-mon[32704]: pgmap v603: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:50.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:50 smithi102 ceph-mon[28094]: pgmap v603: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:51.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:50 smithi112 ceph-mon[32769]: pgmap v603: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:41:52.829 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:52 smithi112 ceph-mon[32769]: pgmap v604: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:52.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:52 smithi130 ceph-mon[32704]: pgmap v604: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:52.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:52 smithi102 ceph-mon[28094]: pgmap v604: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:41:54.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:54 smithi130 ceph-mon[32704]: pgmap v605: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:54.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:54 smithi102 ceph-mon[28094]: pgmap v605: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:55.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:54 smithi112 ceph-mon[32769]: pgmap v605: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:41:56.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:56 smithi130 ceph-mon[32704]: pgmap v606: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:56.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:56 smithi102 ceph-mon[28094]: pgmap v606: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:57.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:56 smithi112 ceph-mon[32769]: pgmap v606: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:41:58.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:41:58 smithi130 ceph-mon[32704]: pgmap v607: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:41:58 smithi102 ceph-mon[28094]: pgmap v607: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:41:59.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:41:58 smithi112 ceph-mon[32769]: pgmap v607: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:00.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:00 smithi130 ceph-mon[32704]: pgmap v608: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:00.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:00 smithi102 ceph-mon[28094]: pgmap v608: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:01.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:00 smithi112 ceph-mon[32769]: pgmap v608: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:02.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:02 smithi130 ceph-mon[32704]: pgmap v609: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:02.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:02 smithi102 ceph-mon[28094]: pgmap v609: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:03.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:02 smithi112 ceph-mon[32769]: pgmap v609: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:04.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:04 smithi130 ceph-mon[32704]: pgmap v610: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:04.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:04 smithi102 ceph-mon[28094]: pgmap v610: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:05.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:04 smithi112 ceph-mon[32769]: pgmap v610: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:06.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:06 smithi130 ceph-mon[32704]: pgmap v611: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:06.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:06 smithi102 ceph-mon[28094]: pgmap v611: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:07.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:06 smithi112 ceph-mon[32769]: pgmap v611: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:08.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:08 smithi102 ceph-mon[28094]: pgmap v612: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:08.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:08 smithi130 ceph-mon[32704]: pgmap v612: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:09.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:08 smithi112 ceph-mon[32769]: pgmap v612: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:10.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:10 smithi130 ceph-mon[32704]: pgmap v613: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:10.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:10 smithi102 ceph-mon[28094]: pgmap v613: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:11.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:10 smithi112 ceph-mon[32769]: pgmap v613: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:12.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:12 smithi130 ceph-mon[32704]: pgmap v614: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:12.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:12 smithi102 ceph-mon[28094]: pgmap v614: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:13.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:12 smithi112 ceph-mon[32769]: pgmap v614: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:14.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:14 smithi130 ceph-mon[32704]: pgmap v615: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:14.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:14 smithi102 ceph-mon[28094]: pgmap v615: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:15.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:14 smithi112 ceph-mon[32769]: pgmap v615: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:16.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:16 smithi130 ceph-mon[32704]: pgmap v616: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:16.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:16 smithi102 ceph-mon[28094]: pgmap v616: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:17.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:16 smithi112 ceph-mon[32769]: pgmap v616: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:18.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:18 smithi130 ceph-mon[32704]: pgmap v617: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:18.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:18 smithi102 ceph-mon[28094]: pgmap v617: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:19.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:18 smithi112 ceph-mon[32769]: pgmap v617: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:20.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:20 smithi130 ceph-mon[32704]: pgmap v618: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:20.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:20 smithi102 ceph-mon[28094]: pgmap v618: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:21.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:20 smithi112 ceph-mon[32769]: pgmap v618: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:22.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:22 smithi130 ceph-mon[32704]: pgmap v619: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:22 smithi102 ceph-mon[28094]: pgmap v619: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:23.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:22 smithi112 ceph-mon[32769]: pgmap v619: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:24.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:24 smithi102 ceph-mon[28094]: pgmap v620: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:25.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:24 smithi112 ceph-mon[32769]: pgmap v620: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:24 smithi130 ceph-mon[32704]: pgmap v620: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:26.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:26 smithi102 ceph-mon[28094]: pgmap v621: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:27.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:26 smithi112 ceph-mon[32769]: pgmap v621: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:27.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:26 smithi130 ceph-mon[32704]: pgmap v621: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:28.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:28 smithi102 ceph-mon[28094]: pgmap v622: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:29.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:28 smithi112 ceph-mon[32769]: pgmap v622: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:29.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:28 smithi130 ceph-mon[32704]: pgmap v622: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:30.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:30 smithi102 ceph-mon[28094]: pgmap v623: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:31.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:30 smithi112 ceph-mon[32769]: pgmap v623: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:31.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:30 smithi130 ceph-mon[32704]: pgmap v623: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:31.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:31 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:42:32.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:31 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:42:32.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:31 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:42:33.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:32 smithi112 ceph-mon[32769]: pgmap v624: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:33.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:32 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:42:33.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:32 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:42:33.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:32 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:42:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:32 smithi130 ceph-mon[32704]: pgmap v624: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:32 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:42:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:32 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:42:33.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:32 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:42:33.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:32 smithi102 ceph-mon[28094]: pgmap v624: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:42:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:42:33.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:42:35.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:34 smithi112 ceph-mon[32769]: pgmap v625: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:34 smithi130 ceph-mon[32704]: pgmap v625: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:35.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:34 smithi102 ceph-mon[28094]: pgmap v625: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:37.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:36 smithi112 ceph-mon[32769]: pgmap v626: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:42:37.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:36 smithi130 ceph-mon[32704]: pgmap v626: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:42:37.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:36 smithi102 ceph-mon[28094]: pgmap v626: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:42:39.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:38 smithi112 ceph-mon[32769]: pgmap v627: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:42:39.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:38 smithi130 ceph-mon[32704]: pgmap v627: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:42:39.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:38 smithi102 ceph-mon[28094]: pgmap v627: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:42:41.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:40 smithi112 ceph-mon[32769]: pgmap v628: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:42:41.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:40 smithi130 ceph-mon[32704]: pgmap v628: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:42:41.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:40 smithi102 ceph-mon[28094]: pgmap v628: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:42:43.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:42 smithi112 ceph-mon[32769]: pgmap v629: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:42:43.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:42 smithi130 ceph-mon[32704]: pgmap v629: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:42:43.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:42 smithi102 ceph-mon[28094]: pgmap v629: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:42:45.030 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:44 smithi112 ceph-mon[32769]: pgmap v630: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:42:45.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:44 smithi130 ceph-mon[32704]: pgmap v630: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:42:45.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:44 smithi102 ceph-mon[28094]: pgmap v630: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:42:47.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:46 smithi112 ceph-mon[32769]: pgmap v631: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:42:47.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:46 smithi130 ceph-mon[32704]: pgmap v631: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:42:47.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:46 smithi102 ceph-mon[28094]: pgmap v631: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:42:49.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:48 smithi112 ceph-mon[32769]: pgmap v632: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:49.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:48 smithi130 ceph-mon[32704]: pgmap v632: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:49.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:48 smithi102 ceph-mon[28094]: pgmap v632: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:51.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:50 smithi112 ceph-mon[32769]: pgmap v633: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:51.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:50 smithi130 ceph-mon[32704]: pgmap v633: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:51.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:50 smithi102 ceph-mon[28094]: pgmap v633: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:42:53.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:52 smithi112 ceph-mon[32769]: pgmap v634: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:53.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:52 smithi130 ceph-mon[32704]: pgmap v634: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:53.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:52 smithi102 ceph-mon[28094]: pgmap v634: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:42:55.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:54 smithi112 ceph-mon[32769]: pgmap v635: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:55.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:54 smithi130 ceph-mon[32704]: pgmap v635: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:55.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:54 smithi102 ceph-mon[28094]: pgmap v635: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:42:57.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:56 smithi112 ceph-mon[32769]: pgmap v636: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:57.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:56 smithi130 ceph-mon[32704]: pgmap v636: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:57.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:56 smithi102 ceph-mon[28094]: pgmap v636: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:42:59.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:42:58 smithi112 ceph-mon[32769]: pgmap v637: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:59.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:42:58 smithi130 ceph-mon[32704]: pgmap v637: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:42:59.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:42:58 smithi102 ceph-mon[28094]: pgmap v637: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:01.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:00 smithi112 ceph-mon[32769]: pgmap v638: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:01.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:00 smithi130 ceph-mon[32704]: pgmap v638: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:01.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:00 smithi102 ceph-mon[28094]: pgmap v638: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:03.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:02 smithi112 ceph-mon[32769]: pgmap v639: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:03.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:02 smithi130 ceph-mon[32704]: pgmap v639: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:03.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:02 smithi102 ceph-mon[28094]: pgmap v639: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:05.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:04 smithi112 ceph-mon[32769]: pgmap v640: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:05.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:04 smithi130 ceph-mon[32704]: pgmap v640: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:05.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:04 smithi102 ceph-mon[28094]: pgmap v640: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:07.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:06 smithi112 ceph-mon[32769]: pgmap v641: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:07.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:06 smithi130 ceph-mon[32704]: pgmap v641: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:07.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:06 smithi102 ceph-mon[28094]: pgmap v641: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:09.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:08 smithi112 ceph-mon[32769]: pgmap v642: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:09.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:08 smithi130 ceph-mon[32704]: pgmap v642: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:09.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:08 smithi102 ceph-mon[28094]: pgmap v642: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:11.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:10 smithi112 ceph-mon[32769]: pgmap v643: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:11.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:10 smithi130 ceph-mon[32704]: pgmap v643: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:11.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:10 smithi102 ceph-mon[28094]: pgmap v643: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:13.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:12 smithi112 ceph-mon[32769]: pgmap v644: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:13.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:12 smithi130 ceph-mon[32704]: pgmap v644: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:13.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:12 smithi102 ceph-mon[28094]: pgmap v644: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:15.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:14 smithi112 ceph-mon[32769]: pgmap v645: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:15.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:14 smithi130 ceph-mon[32704]: pgmap v645: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:15.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:14 smithi102 ceph-mon[28094]: pgmap v645: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:17.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:16 smithi112 ceph-mon[32769]: pgmap v646: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:17.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:16 smithi130 ceph-mon[32704]: pgmap v646: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:17.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:16 smithi102 ceph-mon[28094]: pgmap v646: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:19.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:18 smithi112 ceph-mon[32769]: pgmap v647: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:19.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:18 smithi130 ceph-mon[32704]: pgmap v647: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:19.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:18 smithi102 ceph-mon[28094]: pgmap v647: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:20 smithi112 ceph-mon[32769]: pgmap v648: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:21.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:20 smithi130 ceph-mon[32704]: pgmap v648: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:21.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:20 smithi102 ceph-mon[28094]: pgmap v648: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:23.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:22 smithi130 ceph-mon[32704]: pgmap v649: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:43:23.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:22 smithi102 ceph-mon[28094]: pgmap v649: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:43:23.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:22 smithi112 ceph-mon[32769]: pgmap v649: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:43:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:24 smithi130 ceph-mon[32704]: pgmap v650: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:43:25.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:24 smithi102 ceph-mon[28094]: pgmap v650: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:43:25.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:24 smithi112 ceph-mon[32769]: pgmap v650: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-26T20:43:27.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:26 smithi130 ceph-mon[32704]: pgmap v651: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:43:27.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:26 smithi102 ceph-mon[28094]: pgmap v651: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:43:27.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:26 smithi112 ceph-mon[32769]: pgmap v651: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:43:29.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:28 smithi130 ceph-mon[32704]: pgmap v652: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:43:29.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:28 smithi102 ceph-mon[28094]: pgmap v652: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:43:29.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:28 smithi112 ceph-mon[32769]: pgmap v652: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:43:31.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:30 smithi130 ceph-mon[32704]: pgmap v653: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:43:31.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:30 smithi102 ceph-mon[28094]: pgmap v653: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:43:31.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:30 smithi112 ceph-mon[32769]: pgmap v653: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:43:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:32 smithi130 ceph-mon[32704]: pgmap v654: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:43:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:32 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:43:33.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:32 smithi102 ceph-mon[28094]: pgmap v654: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:43:33.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:32 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:43:33.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:32 smithi112 ceph-mon[32769]: pgmap v654: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:43:33.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:32 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:43:33.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:43:33.411+0000 7f8060c00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f8040005b70 con 0x7f80641028c0 2024-04-26T20:43:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:43:33.411+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f8040005b70 con 0x7f80641028c0 2024-04-26T20:43:34.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:43:34.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:43:34.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:33 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:43:34.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:43:34.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:43:34.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:33 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:43:34.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:43:34.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:43:34.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:33 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:43:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:34 smithi130 ceph-mon[32704]: pgmap v655: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:35.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:34 smithi102 ceph-mon[28094]: pgmap v655: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:35.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:34 smithi112 ceph-mon[32769]: pgmap v655: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:36.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:35 smithi130 ceph-mon[32704]: pgmap v656: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:36.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:35 smithi102 ceph-mon[28094]: pgmap v656: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:36.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:35 smithi112 ceph-mon[32769]: pgmap v656: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:38.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:37 smithi112 ceph-mon[32769]: pgmap v657: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:38.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:37 smithi130 ceph-mon[32704]: pgmap v657: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:38.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:37 smithi102 ceph-mon[28094]: pgmap v657: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:40.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:39 smithi112 ceph-mon[32769]: pgmap v658: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:40.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:39 smithi130 ceph-mon[32704]: pgmap v658: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:40.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:39 smithi102 ceph-mon[28094]: pgmap v658: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:42.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:41 smithi112 ceph-mon[32769]: pgmap v659: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:42.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:41 smithi130 ceph-mon[32704]: pgmap v659: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:42.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:41 smithi102 ceph-mon[28094]: pgmap v659: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:44.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:43 smithi112 ceph-mon[32769]: pgmap v660: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:44.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:43 smithi130 ceph-mon[32704]: pgmap v660: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:44.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:43 smithi102 ceph-mon[28094]: pgmap v660: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:46.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:45 smithi112 ceph-mon[32769]: pgmap v661: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:46.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:45 smithi130 ceph-mon[32704]: pgmap v661: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:46.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:45 smithi102 ceph-mon[28094]: pgmap v661: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:48.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:47 smithi112 ceph-mon[32769]: pgmap v662: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:48.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:47 smithi130 ceph-mon[32704]: pgmap v662: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:48.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:47 smithi102 ceph-mon[28094]: pgmap v662: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:50.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:49 smithi112 ceph-mon[32769]: pgmap v663: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:50.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:49 smithi130 ceph-mon[32704]: pgmap v663: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:50.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:49 smithi102 ceph-mon[28094]: pgmap v663: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:43:52.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:51 smithi112 ceph-mon[32769]: pgmap v664: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:52.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:51 smithi130 ceph-mon[32704]: pgmap v664: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:52.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:51 smithi102 ceph-mon[28094]: pgmap v664: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:43:54.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:53 smithi112 ceph-mon[32769]: pgmap v665: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:54.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:53 smithi130 ceph-mon[32704]: pgmap v665: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:54.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:53 smithi102 ceph-mon[28094]: pgmap v665: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:43:56.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:55 smithi112 ceph-mon[32769]: pgmap v666: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:56.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:55 smithi130 ceph-mon[32704]: pgmap v666: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:56.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:55 smithi102 ceph-mon[28094]: pgmap v666: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:43:58.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:57 smithi112 ceph-mon[32769]: pgmap v667: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:58.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:57 smithi130 ceph-mon[32704]: pgmap v667: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:43:58.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:57 smithi102 ceph-mon[28094]: pgmap v667: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:00.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:43:59 smithi112 ceph-mon[32769]: pgmap v668: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:00.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:43:59 smithi130 ceph-mon[32704]: pgmap v668: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:00.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:43:59 smithi102 ceph-mon[28094]: pgmap v668: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:02.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:01 smithi112 ceph-mon[32769]: pgmap v669: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:02.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:01 smithi130 ceph-mon[32704]: pgmap v669: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:02.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:01 smithi102 ceph-mon[28094]: pgmap v669: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:04.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:03 smithi112 ceph-mon[32769]: pgmap v670: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:04.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:03 smithi130 ceph-mon[32704]: pgmap v670: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:04.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:03 smithi102 ceph-mon[28094]: pgmap v670: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:06.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:05 smithi112 ceph-mon[32769]: pgmap v671: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:06.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:05 smithi130 ceph-mon[32704]: pgmap v671: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:06.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:05 smithi102 ceph-mon[28094]: pgmap v671: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:08.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:08 smithi130 ceph-mon[32704]: pgmap v672: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:08.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:08 smithi102 ceph-mon[28094]: pgmap v672: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:08.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:08 smithi112 ceph-mon[32769]: pgmap v672: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:10.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:10 smithi130 ceph-mon[32704]: pgmap v673: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:10.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:10 smithi102 ceph-mon[28094]: pgmap v673: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:10.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:10 smithi112 ceph-mon[32769]: pgmap v673: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:12.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:12 smithi102 ceph-mon[28094]: pgmap v674: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:12.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:12 smithi112 ceph-mon[32769]: pgmap v674: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:12.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:12 smithi130 ceph-mon[32704]: pgmap v674: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:14.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:14 smithi102 ceph-mon[28094]: pgmap v675: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:14.673 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:14 smithi130 ceph-mon[32704]: pgmap v675: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:14.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:14 smithi112 ceph-mon[32769]: pgmap v675: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:16.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:16 smithi102 ceph-mon[28094]: pgmap v676: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:16.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:16 smithi112 ceph-mon[32769]: pgmap v676: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:16.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:16 smithi130 ceph-mon[32704]: pgmap v676: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:18.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:44:18.419+0000 7f8060c00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f8040005ef0 con 0x7f80641028c0 2024-04-26T20:44:18.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:44:18.419+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f8040005ef0 con 0x7f80641028c0 2024-04-26T20:44:18.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:18 smithi102 ceph-mon[28094]: pgmap v677: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:18.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:18 smithi112 ceph-mon[32769]: pgmap v677: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:18.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:18 smithi130 ceph-mon[32704]: pgmap v677: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:20.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:20 smithi102 ceph-mon[28094]: pgmap v678: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:20.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:20 smithi112 ceph-mon[32769]: pgmap v678: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:20.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:20 smithi130 ceph-mon[32704]: pgmap v678: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:22.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:22 smithi112 ceph-mon[32769]: pgmap v679: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:22.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:22 smithi130 ceph-mon[32704]: pgmap v679: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:22 smithi102 ceph-mon[28094]: pgmap v679: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:24.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:24 smithi112 ceph-mon[32769]: pgmap v680: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:24.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:24 smithi130 ceph-mon[32704]: pgmap v680: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:24.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:24 smithi102 ceph-mon[28094]: pgmap v680: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:26.701 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:26 smithi102 ceph-mon[28094]: pgmap v681: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:26.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:26 smithi112 ceph-mon[32769]: pgmap v681: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:26.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:26 smithi130 ceph-mon[32704]: pgmap v681: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:28.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:28 smithi112 ceph-mon[32769]: pgmap v682: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:28.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:28 smithi130 ceph-mon[32704]: pgmap v682: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:28.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:28 smithi102 ceph-mon[28094]: pgmap v682: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:30.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:30 smithi112 ceph-mon[32769]: pgmap v683: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:30.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:30 smithi130 ceph-mon[32704]: pgmap v683: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:30.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:30 smithi102 ceph-mon[28094]: pgmap v683: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:32.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:32 smithi112 ceph-mon[32769]: pgmap v684: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:32.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:32 smithi130 ceph-mon[32704]: pgmap v684: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:32.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:32 smithi102 ceph-mon[28094]: pgmap v684: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:34.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:34 smithi112 ceph-mon[32769]: pgmap v685: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:34.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:34 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:44:34.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:34 smithi130 ceph-mon[32704]: pgmap v685: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:34.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:34 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:44:34.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:34 smithi102 ceph-mon[28094]: pgmap v685: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-26T20:44:34.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:34 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:44:35.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:44:35.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:44:35.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:44:35.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:44:35.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:44:35.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:44:35.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:44:35.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:44:35.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:44:36.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:36 smithi112 ceph-mon[32769]: pgmap v686: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:36.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:36 smithi130 ceph-mon[32704]: pgmap v686: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:36.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:36 smithi102 ceph-mon[28094]: pgmap v686: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:38.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:38 smithi112 ceph-mon[32769]: pgmap v687: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:38.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:38 smithi130 ceph-mon[32704]: pgmap v687: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:38.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:38 smithi102 ceph-mon[28094]: pgmap v687: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:40.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:40 smithi112 ceph-mon[32769]: pgmap v688: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:40.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:40 smithi130 ceph-mon[32704]: pgmap v688: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:40.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:40 smithi102 ceph-mon[28094]: pgmap v688: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:42.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:42 smithi112 ceph-mon[32769]: pgmap v689: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:42.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:42 smithi130 ceph-mon[32704]: pgmap v689: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:42.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:42 smithi102 ceph-mon[28094]: pgmap v689: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:44.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:44 smithi112 ceph-mon[32769]: pgmap v690: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-26T20:44:44.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:44 smithi130 ceph-mon[32704]: pgmap v690: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-26T20:44:44.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:44 smithi102 ceph-mon[28094]: pgmap v690: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-26T20:44:46.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:46 smithi112 ceph-mon[32769]: pgmap v691: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:46.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:46 smithi130 ceph-mon[32704]: pgmap v691: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:46.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:46 smithi102 ceph-mon[28094]: pgmap v691: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-04-26T20:44:48.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:48 smithi112 ceph-mon[32769]: pgmap v692: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:48.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:48 smithi130 ceph-mon[32704]: pgmap v692: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:48.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:48 smithi102 ceph-mon[28094]: pgmap v692: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:50.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:50 smithi112 ceph-mon[32769]: pgmap v693: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:50.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:50 smithi130 ceph-mon[32704]: pgmap v693: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:50.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:50 smithi102 ceph-mon[28094]: pgmap v693: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:44:52.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:52 smithi112 ceph-mon[32769]: pgmap v694: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:52.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:52 smithi130 ceph-mon[32704]: pgmap v694: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:52.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:52 smithi102 ceph-mon[28094]: pgmap v694: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:44:54.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:54 smithi112 ceph-mon[32769]: pgmap v695: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-26T20:44:54.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:54 smithi130 ceph-mon[32704]: pgmap v695: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-26T20:44:54.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:54 smithi102 ceph-mon[28094]: pgmap v695: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-26T20:44:56.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:56 smithi112 ceph-mon[32769]: pgmap v696: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-26T20:44:56.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:56 smithi130 ceph-mon[32704]: pgmap v696: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-26T20:44:56.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:56 smithi102 ceph-mon[28094]: pgmap v696: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-26T20:44:58.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:44:58 smithi112 ceph-mon[32769]: pgmap v697: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:58.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:44:58 smithi130 ceph-mon[32704]: pgmap v697: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:44:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:44:58 smithi102 ceph-mon[28094]: pgmap v697: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:00.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:00 smithi112 ceph-mon[32769]: pgmap v698: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:00.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:00 smithi130 ceph-mon[32704]: pgmap v698: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:00.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:00 smithi102 ceph-mon[28094]: pgmap v698: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:02.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:02 smithi112 ceph-mon[32769]: pgmap v699: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:02.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:02 smithi130 ceph-mon[32704]: pgmap v699: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:02.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:02 smithi102 ceph-mon[28094]: pgmap v699: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:04.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:04 smithi112 ceph-mon[32769]: pgmap v700: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-26T20:45:04.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:04 smithi130 ceph-mon[32704]: pgmap v700: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-26T20:45:04.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:04 smithi102 ceph-mon[28094]: pgmap v700: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-26T20:45:06.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:06 smithi112 ceph-mon[32769]: pgmap v701: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-26T20:45:06.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:06 smithi130 ceph-mon[32704]: pgmap v701: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-26T20:45:06.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:06 smithi102 ceph-mon[28094]: pgmap v701: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-26T20:45:08.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:08 smithi112 ceph-mon[32769]: pgmap v702: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:08.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:08 smithi130 ceph-mon[32704]: pgmap v702: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:08.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:08 smithi102 ceph-mon[28094]: pgmap v702: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:10.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:10 smithi130 ceph-mon[32704]: pgmap v703: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:10.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:10 smithi102 ceph-mon[28094]: pgmap v703: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:11.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:10 smithi112 ceph-mon[32769]: pgmap v703: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:12.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:12 smithi130 ceph-mon[32704]: pgmap v704: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:12.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:12 smithi102 ceph-mon[28094]: pgmap v704: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:13.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:12 smithi112 ceph-mon[32769]: pgmap v704: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:14.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:14 smithi130 ceph-mon[32704]: pgmap v705: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:14.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:14 smithi102 ceph-mon[28094]: pgmap v705: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:15.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:14 smithi112 ceph-mon[32769]: pgmap v705: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:16.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:16 smithi130 ceph-mon[32704]: pgmap v706: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-26T20:45:16.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:16 smithi102 ceph-mon[28094]: pgmap v706: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-26T20:45:17.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:16 smithi112 ceph-mon[32769]: pgmap v706: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-26T20:45:18.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:18 smithi130 ceph-mon[32704]: pgmap v707: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:18.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:18 smithi102 ceph-mon[28094]: pgmap v707: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:19.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:18 smithi112 ceph-mon[32769]: pgmap v707: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:20.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:20 smithi130 ceph-mon[32704]: pgmap v708: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:20.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:20 smithi102 ceph-mon[28094]: pgmap v708: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:21.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:20 smithi112 ceph-mon[32769]: pgmap v708: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:22.829 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:22 smithi112 ceph-mon[32769]: pgmap v709: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:22.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:22 smithi130 ceph-mon[32704]: pgmap v709: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:22.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:22 smithi102 ceph-mon[28094]: pgmap v709: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:24.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:24 smithi130 ceph-mon[32704]: pgmap v710: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:24.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:24 smithi102 ceph-mon[28094]: pgmap v710: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:25.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:24 smithi112 ceph-mon[32769]: pgmap v710: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:26.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:26 smithi130 ceph-mon[32704]: pgmap v711: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:26.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:26 smithi102 ceph-mon[28094]: pgmap v711: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:27.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:26 smithi112 ceph-mon[32769]: pgmap v711: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:28.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:28 smithi130 ceph-mon[32704]: pgmap v712: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:28.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:28 smithi102 ceph-mon[28094]: pgmap v712: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:29.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:28 smithi112 ceph-mon[32769]: pgmap v712: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:30.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:30 smithi130 ceph-mon[32704]: pgmap v713: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:30.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:30 smithi102 ceph-mon[28094]: pgmap v713: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:31.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:30 smithi112 ceph-mon[32769]: pgmap v713: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:32.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:32 smithi130 ceph-mon[32704]: pgmap v714: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:32.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:32 smithi102 ceph-mon[28094]: pgmap v714: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:33.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:32 smithi112 ceph-mon[32769]: pgmap v714: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:34.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:34 smithi130 ceph-mon[32704]: pgmap v715: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:34.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:34 smithi102 ceph-mon[28094]: pgmap v715: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:35.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:34 smithi112 ceph-mon[32769]: pgmap v715: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:35.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:35 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:45:35.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:35 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:45:36.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:35 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:45:36.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:36 smithi130 ceph-mon[32704]: pgmap v716: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:36.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:36 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.881 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:36 smithi112 ceph-mon[32769]: pgmap v716: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:36.882 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:36 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.882 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:36 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.882 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:36 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.882 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:36 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.882 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:36 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.882 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:36 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:36 smithi102 ceph-mon[28094]: pgmap v716: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:36.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:36.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:36 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:38.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:38 smithi130 ceph-mon[32704]: pgmap v717: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:38.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:38.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:38.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:38.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:38 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:38.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:38 smithi102 ceph-mon[28094]: pgmap v717: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:38.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:38.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:38.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:38.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:38 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:39.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:38 smithi112 ceph-mon[32769]: pgmap v717: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:39.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:39.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:39.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:39.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:38 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:40.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:40 smithi112 ceph-mon[32769]: pgmap v718: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:40.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:40.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:40 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:40.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:40 smithi130 ceph-mon[32704]: pgmap v718: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:40.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:40.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:40 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:40.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:40 smithi102 ceph-mon[28094]: pgmap v718: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:40.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:40.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:40 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:41.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:41 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:45:41.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:41 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:45:41.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:41 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:41.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:41 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:45:41.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:41 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:45:41.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:41 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:41.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:41 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:45:41.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:41 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:45:41.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:41 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:45:42.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:42 smithi130 ceph-mon[32704]: pgmap v719: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:42.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:42 smithi102 ceph-mon[28094]: pgmap v719: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:43.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:42 smithi112 ceph-mon[32769]: pgmap v719: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:44.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:44 smithi130 ceph-mon[32704]: pgmap v720: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:44.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:44 smithi102 ceph-mon[28094]: pgmap v720: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:45.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:44 smithi112 ceph-mon[32769]: pgmap v720: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:46.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:46 smithi130 ceph-mon[32704]: pgmap v721: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:46.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:46 smithi102 ceph-mon[28094]: pgmap v721: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:47.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:46 smithi112 ceph-mon[32769]: pgmap v721: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:48.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:48 smithi130 ceph-mon[32704]: pgmap v722: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:48.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:48 smithi102 ceph-mon[28094]: pgmap v722: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:49.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:48 smithi112 ceph-mon[32769]: pgmap v722: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:50.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:50 smithi130 ceph-mon[32704]: pgmap v723: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:50.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:50 smithi102 ceph-mon[28094]: pgmap v723: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:51.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:50 smithi112 ceph-mon[32769]: pgmap v723: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:45:52.846 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:52 smithi112 ceph-mon[32769]: pgmap v724: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:52.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:52 smithi130 ceph-mon[32704]: pgmap v724: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:52.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:52 smithi102 ceph-mon[28094]: pgmap v724: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:54.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:54 smithi130 ceph-mon[32704]: pgmap v725: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:54.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:54 smithi102 ceph-mon[28094]: pgmap v725: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:55.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:54 smithi112 ceph-mon[32769]: pgmap v725: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:45:56.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:56 smithi130 ceph-mon[32704]: pgmap v726: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:56.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:56 smithi102 ceph-mon[28094]: pgmap v726: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:57.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:56 smithi112 ceph-mon[32769]: pgmap v726: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:45:58.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:45:58 smithi130 ceph-mon[32704]: pgmap v727: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:58.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:45:58 smithi102 ceph-mon[28094]: pgmap v727: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:45:59.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:45:58 smithi112 ceph-mon[32769]: pgmap v727: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:00.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:00 smithi130 ceph-mon[32704]: pgmap v728: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:00.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:00 smithi102 ceph-mon[28094]: pgmap v728: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:01.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:00 smithi112 ceph-mon[32769]: pgmap v728: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:02.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:02 smithi130 ceph-mon[32704]: pgmap v729: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:02.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:02 smithi102 ceph-mon[28094]: pgmap v729: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:03.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:02 smithi112 ceph-mon[32769]: pgmap v729: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:04.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:04 smithi130 ceph-mon[32704]: pgmap v730: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:04.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:04 smithi102 ceph-mon[28094]: pgmap v730: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:05.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:04 smithi112 ceph-mon[32769]: pgmap v730: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:06.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:06 smithi130 ceph-mon[32704]: pgmap v731: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:06.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:06 smithi102 ceph-mon[28094]: pgmap v731: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:07.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:06 smithi112 ceph-mon[32769]: pgmap v731: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:08.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:08 smithi130 ceph-mon[32704]: pgmap v732: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:08.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:08 smithi102 ceph-mon[28094]: pgmap v732: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:09.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:08 smithi112 ceph-mon[32769]: pgmap v732: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:10.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:10 smithi102 ceph-mon[28094]: pgmap v733: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:11.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:10 smithi112 ceph-mon[32769]: pgmap v733: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:11.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:10 smithi130 ceph-mon[32704]: pgmap v733: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:12.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:12 smithi102 ceph-mon[28094]: pgmap v734: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:13.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:12 smithi112 ceph-mon[32769]: pgmap v734: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:13.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:12 smithi130 ceph-mon[32704]: pgmap v734: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:14.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:14 smithi102 ceph-mon[28094]: pgmap v735: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:14.923 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:14 smithi130 ceph-mon[32704]: pgmap v735: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:15.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:14 smithi112 ceph-mon[32769]: pgmap v735: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:16.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:16 smithi102 ceph-mon[28094]: pgmap v736: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:17.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:16 smithi112 ceph-mon[32769]: pgmap v736: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:17.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:16 smithi130 ceph-mon[32704]: pgmap v736: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:18.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:18 smithi102 ceph-mon[28094]: pgmap v737: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:19.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:18 smithi112 ceph-mon[32769]: pgmap v737: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:19.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:18 smithi130 ceph-mon[32704]: pgmap v737: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:21.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:20 smithi112 ceph-mon[32769]: pgmap v738: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:21.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:20 smithi130 ceph-mon[32704]: pgmap v738: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:21.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:20 smithi102 ceph-mon[28094]: pgmap v738: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:23.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:22 smithi112 ceph-mon[32769]: pgmap v739: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:23.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:22 smithi130 ceph-mon[32704]: pgmap v739: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:23.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:22 smithi102 ceph-mon[28094]: pgmap v739: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:25.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:24 smithi112 ceph-mon[32769]: pgmap v740: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:24 smithi130 ceph-mon[32704]: pgmap v740: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:25.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:24 smithi102 ceph-mon[28094]: pgmap v740: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:26.950 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:26 smithi102 ceph-mon[28094]: pgmap v741: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:27.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:26 smithi112 ceph-mon[32769]: pgmap v741: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:27.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:26 smithi130 ceph-mon[32704]: pgmap v741: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:29.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:28 smithi112 ceph-mon[32769]: pgmap v742: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:29.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:28 smithi130 ceph-mon[32704]: pgmap v742: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:29.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:28 smithi102 ceph-mon[28094]: pgmap v742: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:31.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:30 smithi112 ceph-mon[32769]: pgmap v743: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:31.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:30 smithi130 ceph-mon[32704]: pgmap v743: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:31.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:30 smithi102 ceph-mon[28094]: pgmap v743: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:33.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:32 smithi112 ceph-mon[32769]: pgmap v744: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:32 smithi130 ceph-mon[32704]: pgmap v744: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:33.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:32 smithi102 ceph-mon[28094]: pgmap v744: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:35.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:34 smithi112 ceph-mon[32769]: pgmap v745: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:34 smithi130 ceph-mon[32704]: pgmap v745: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:35.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:34 smithi102 ceph-mon[28094]: pgmap v745: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:37.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:36 smithi112 ceph-mon[32769]: pgmap v746: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:37.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:36 smithi130 ceph-mon[32704]: pgmap v746: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:37.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:36 smithi102 ceph-mon[28094]: pgmap v746: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:39.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:38 smithi112 ceph-mon[32769]: pgmap v747: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:39.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:38 smithi130 ceph-mon[32704]: pgmap v747: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:39.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:38 smithi102 ceph-mon[28094]: pgmap v747: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:41.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:40 smithi112 ceph-mon[32769]: pgmap v748: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:41.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:40 smithi130 ceph-mon[32704]: pgmap v748: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:41.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:40 smithi102 ceph-mon[28094]: pgmap v748: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:42.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:41 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:46:42.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:41 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:46:42.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:41 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:46:43.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:42 smithi112 ceph-mon[32769]: pgmap v749: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:43.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:42 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:46:43.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:42 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:46:43.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:42 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:46:43.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:42 smithi130 ceph-mon[32704]: pgmap v749: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:43.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:42 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:46:43.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:42 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:46:43.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:42 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:46:43.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:42 smithi102 ceph-mon[28094]: pgmap v749: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:43.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:42 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:46:43.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:42 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:46:43.149 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:42 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:46:45.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:44 smithi112 ceph-mon[32769]: pgmap v750: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:45.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:44 smithi130 ceph-mon[32704]: pgmap v750: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:45.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:44 smithi102 ceph-mon[28094]: pgmap v750: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:47.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:46 smithi112 ceph-mon[32769]: pgmap v751: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:47.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:46 smithi130 ceph-mon[32704]: pgmap v751: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:47.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:46 smithi102 ceph-mon[28094]: pgmap v751: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:49.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:48 smithi112 ceph-mon[32769]: pgmap v752: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:49.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:48 smithi130 ceph-mon[32704]: pgmap v752: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:49.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:48 smithi102 ceph-mon[28094]: pgmap v752: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:51.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:50 smithi112 ceph-mon[32769]: pgmap v753: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:51.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:50 smithi130 ceph-mon[32704]: pgmap v753: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:51.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:50 smithi102 ceph-mon[28094]: pgmap v753: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:46:53.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:52 smithi112 ceph-mon[32769]: pgmap v754: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:53.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:52 smithi130 ceph-mon[32704]: pgmap v754: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:53.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:52 smithi102 ceph-mon[28094]: pgmap v754: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:55.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:54 smithi112 ceph-mon[32769]: pgmap v755: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:55.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:54 smithi130 ceph-mon[32704]: pgmap v755: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:55.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:54 smithi102 ceph-mon[28094]: pgmap v755: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:46:57.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:56 smithi112 ceph-mon[32769]: pgmap v756: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:57.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:56 smithi130 ceph-mon[32704]: pgmap v756: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:57.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:56 smithi102 ceph-mon[28094]: pgmap v756: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:46:59.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:46:58 smithi112 ceph-mon[32769]: pgmap v757: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:59.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:46:58 smithi130 ceph-mon[32704]: pgmap v757: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:46:59.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:46:58 smithi102 ceph-mon[28094]: pgmap v757: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:01.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:00 smithi112 ceph-mon[32769]: pgmap v758: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:01.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:00 smithi130 ceph-mon[32704]: pgmap v758: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:01.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:00 smithi102 ceph-mon[28094]: pgmap v758: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:03.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:02 smithi112 ceph-mon[32769]: pgmap v759: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:03.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:02 smithi130 ceph-mon[32704]: pgmap v759: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:03.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:02 smithi102 ceph-mon[28094]: pgmap v759: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:05.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:04 smithi112 ceph-mon[32769]: pgmap v760: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:05.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:04 smithi130 ceph-mon[32704]: pgmap v760: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:05.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:04 smithi102 ceph-mon[28094]: pgmap v760: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:07.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:06 smithi130 ceph-mon[32704]: pgmap v761: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:07.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:06 smithi102 ceph-mon[28094]: pgmap v761: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:07.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:06 smithi112 ceph-mon[32769]: pgmap v761: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:09.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:08 smithi130 ceph-mon[32704]: pgmap v762: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:09.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:08 smithi102 ceph-mon[28094]: pgmap v762: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:09.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:08 smithi112 ceph-mon[32769]: pgmap v762: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:11.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:10 smithi130 ceph-mon[32704]: pgmap v763: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:11.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:10 smithi102 ceph-mon[28094]: pgmap v763: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:11.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:10 smithi112 ceph-mon[32769]: pgmap v763: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:13.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:12 smithi130 ceph-mon[32704]: pgmap v764: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:13.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:12 smithi102 ceph-mon[28094]: pgmap v764: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:13.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:12 smithi112 ceph-mon[32769]: pgmap v764: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:15.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:14 smithi130 ceph-mon[32704]: pgmap v765: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:15.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:14 smithi102 ceph-mon[28094]: pgmap v765: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:15.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:14 smithi112 ceph-mon[32769]: pgmap v765: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:17.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:16 smithi130 ceph-mon[32704]: pgmap v766: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:17.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:16 smithi102 ceph-mon[28094]: pgmap v766: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:17.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:16 smithi112 ceph-mon[32769]: pgmap v766: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:47:18.448+0000 7f8060c00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f8040006110 con 0x7f80641028c0 2024-04-26T20:47:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:47:18.449+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f8040006110 con 0x7f80641028c0 2024-04-26T20:47:19.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:18 smithi130 ceph-mon[32704]: pgmap v767: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:19.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:18 smithi102 ceph-mon[28094]: pgmap v767: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:19.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:18 smithi112 ceph-mon[32769]: pgmap v767: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:21.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:20 smithi130 ceph-mon[32704]: pgmap v768: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:21.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:20 smithi102 ceph-mon[28094]: pgmap v768: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:21.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:20 smithi112 ceph-mon[32769]: pgmap v768: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:23.122 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:22 smithi130 ceph-mon[32704]: pgmap v769: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:23.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:22 smithi102 ceph-mon[28094]: pgmap v769: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:23.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:22 smithi112 ceph-mon[32769]: pgmap v769: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:25.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:24 smithi130 ceph-mon[32704]: pgmap v770: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:25.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:24 smithi102 ceph-mon[28094]: pgmap v770: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:25.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:24 smithi112 ceph-mon[32769]: pgmap v770: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:27.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:26 smithi130 ceph-mon[32704]: pgmap v771: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:27.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:26 smithi102 ceph-mon[28094]: pgmap v771: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:27.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:26 smithi112 ceph-mon[32769]: pgmap v771: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:29.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:28 smithi130 ceph-mon[32704]: pgmap v772: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:29.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:28 smithi102 ceph-mon[28094]: pgmap v772: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:29.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:28 smithi112 ceph-mon[32769]: pgmap v772: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:31.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:30 smithi130 ceph-mon[32704]: pgmap v773: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:31.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:30 smithi102 ceph-mon[28094]: pgmap v773: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:31.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:30 smithi112 ceph-mon[32769]: pgmap v773: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:32 smithi130 ceph-mon[32704]: pgmap v774: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:33.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:32 smithi102 ceph-mon[28094]: pgmap v774: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:33.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:32 smithi112 ceph-mon[32769]: pgmap v774: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:35.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:34 smithi130 ceph-mon[32704]: pgmap v775: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:35.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:34 smithi102 ceph-mon[28094]: pgmap v775: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:35.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:34 smithi112 ceph-mon[32769]: pgmap v775: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:36.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:35 smithi112 ceph-mon[32769]: pgmap v776: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:36.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:35 smithi130 ceph-mon[32704]: pgmap v776: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:36.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:35 smithi102 ceph-mon[28094]: pgmap v776: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:38.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:38 smithi130 ceph-mon[32704]: pgmap v777: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:38.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:38 smithi102 ceph-mon[28094]: pgmap v777: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:38.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:38 smithi112 ceph-mon[32769]: pgmap v777: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:40.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:40 smithi130 ceph-mon[32704]: pgmap v778: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:40.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:40 smithi102 ceph-mon[28094]: pgmap v778: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:40.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:40 smithi112 ceph-mon[32769]: pgmap v778: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:42.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:42 smithi130 ceph-mon[32704]: pgmap v779: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:42.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:42 smithi102 ceph-mon[28094]: pgmap v779: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:42.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:42 smithi112 ceph-mon[32769]: pgmap v779: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:43.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:43 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:47:43.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:43 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:47:43.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:43 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:47:44.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:44 smithi130 ceph-mon[32704]: pgmap v780: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:44.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:47:44.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:47:44.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:47:44.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:44 smithi102 ceph-mon[28094]: pgmap v780: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:44.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:47:44.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:47:44.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:47:44.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:44 smithi112 ceph-mon[32769]: pgmap v780: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:44.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:47:44.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:47:44.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:47:46.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:46 smithi130 ceph-mon[32704]: pgmap v781: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:46.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:46 smithi102 ceph-mon[28094]: pgmap v781: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:46.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:46 smithi112 ceph-mon[32769]: pgmap v781: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:48.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:48 smithi130 ceph-mon[32704]: pgmap v782: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:48.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:48 smithi102 ceph-mon[28094]: pgmap v782: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:48.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:48 smithi112 ceph-mon[32769]: pgmap v782: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:50.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:50 smithi130 ceph-mon[32704]: pgmap v783: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:50.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:50 smithi102 ceph-mon[28094]: pgmap v783: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:50.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:50 smithi112 ceph-mon[32769]: pgmap v783: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:47:52.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:52 smithi130 ceph-mon[32704]: pgmap v784: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:52.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:52 smithi102 ceph-mon[28094]: pgmap v784: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:52.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:52 smithi112 ceph-mon[32769]: pgmap v784: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:54.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:54 smithi130 ceph-mon[32704]: pgmap v785: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:54.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:54 smithi102 ceph-mon[28094]: pgmap v785: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:54.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:54 smithi112 ceph-mon[32769]: pgmap v785: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:47:56.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:56 smithi130 ceph-mon[32704]: pgmap v786: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:56.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:56 smithi102 ceph-mon[28094]: pgmap v786: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:56.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:56 smithi112 ceph-mon[32769]: pgmap v786: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:47:58.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:47:58 smithi130 ceph-mon[32704]: pgmap v787: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:58.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:47:58 smithi102 ceph-mon[28094]: pgmap v787: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:47:58.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:47:58 smithi112 ceph-mon[32769]: pgmap v787: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:00.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:00 smithi130 ceph-mon[32704]: pgmap v788: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:00.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:00 smithi102 ceph-mon[28094]: pgmap v788: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:00.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:00 smithi112 ceph-mon[32769]: pgmap v788: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:02.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:02 smithi130 ceph-mon[32704]: pgmap v789: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:02.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:02 smithi102 ceph-mon[28094]: pgmap v789: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:02.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:02 smithi112 ceph-mon[32769]: pgmap v789: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:04.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:04 smithi130 ceph-mon[32704]: pgmap v790: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:04.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:04 smithi102 ceph-mon[28094]: pgmap v790: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:04.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:04 smithi112 ceph-mon[32769]: pgmap v790: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:06.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:06 smithi130 ceph-mon[32704]: pgmap v791: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:06.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:06 smithi102 ceph-mon[28094]: pgmap v791: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:06.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:06 smithi112 ceph-mon[32769]: pgmap v791: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:08.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:08 smithi102 ceph-mon[28094]: pgmap v792: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:08.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:08 smithi112 ceph-mon[32769]: pgmap v792: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:08.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:08 smithi130 ceph-mon[32704]: pgmap v792: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:10.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:10 smithi102 ceph-mon[28094]: pgmap v793: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:10.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:10 smithi112 ceph-mon[32769]: pgmap v793: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:10.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:10 smithi130 ceph-mon[32704]: pgmap v793: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:12.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:12 smithi102 ceph-mon[28094]: pgmap v794: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:12.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:12 smithi112 ceph-mon[32769]: pgmap v794: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:12.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:12 smithi130 ceph-mon[32704]: pgmap v794: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:14.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:14 smithi102 ceph-mon[28094]: pgmap v795: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:14.423 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:14 smithi130 ceph-mon[32704]: pgmap v795: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:14.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:14 smithi112 ceph-mon[32769]: pgmap v795: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:16.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:16 smithi102 ceph-mon[28094]: pgmap v796: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:16.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:16 smithi112 ceph-mon[32769]: pgmap v796: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:16.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:16 smithi130 ceph-mon[32704]: pgmap v796: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:18.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:18 smithi112 ceph-mon[32769]: pgmap v797: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:18.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:18 smithi130 ceph-mon[32704]: pgmap v797: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:18.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:18 smithi102 ceph-mon[28094]: pgmap v797: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:20.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:20 smithi112 ceph-mon[32769]: pgmap v798: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:20.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:20 smithi130 ceph-mon[32704]: pgmap v798: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:20.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:20 smithi102 ceph-mon[28094]: pgmap v798: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:22.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:22 smithi112 ceph-mon[32769]: pgmap v799: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:22.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:22 smithi130 ceph-mon[32704]: pgmap v799: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:22.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:22 smithi102 ceph-mon[28094]: pgmap v799: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:24.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:24 smithi112 ceph-mon[32769]: pgmap v800: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:24.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:24 smithi130 ceph-mon[32704]: pgmap v800: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:24.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:24 smithi102 ceph-mon[28094]: pgmap v800: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:26.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:26 smithi102 ceph-mon[28094]: pgmap v801: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:26.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:26 smithi112 ceph-mon[32769]: pgmap v801: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:26.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:26 smithi130 ceph-mon[32704]: pgmap v801: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:28.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:28 smithi112 ceph-mon[32769]: pgmap v802: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:28.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:28 smithi130 ceph-mon[32704]: pgmap v802: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:28.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:28 smithi102 ceph-mon[28094]: pgmap v802: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:30.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:30 smithi112 ceph-mon[32769]: pgmap v803: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:30.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:30 smithi130 ceph-mon[32704]: pgmap v803: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:30.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:30 smithi102 ceph-mon[28094]: pgmap v803: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:32.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:32 smithi112 ceph-mon[32769]: pgmap v804: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:32.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:32 smithi130 ceph-mon[32704]: pgmap v804: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:32.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:32 smithi102 ceph-mon[28094]: pgmap v804: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:34.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:34 smithi112 ceph-mon[32769]: pgmap v805: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:34.626 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:34 smithi130 ceph-mon[32704]: pgmap v805: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:34.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:34 smithi102 ceph-mon[28094]: pgmap v805: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:36.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:36 smithi112 ceph-mon[32769]: pgmap v806: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:36.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:36 smithi130 ceph-mon[32704]: pgmap v806: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:36.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:36 smithi102 ceph-mon[28094]: pgmap v806: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:38.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:38 smithi112 ceph-mon[32769]: pgmap v807: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:38.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:38 smithi130 ceph-mon[32704]: pgmap v807: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:38.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:38 smithi102 ceph-mon[28094]: pgmap v807: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:40.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:40 smithi112 ceph-mon[32769]: pgmap v808: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:40.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:40 smithi130 ceph-mon[32704]: pgmap v808: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:40.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:40 smithi102 ceph-mon[28094]: pgmap v808: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:48:42.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:42 smithi112 ceph-mon[32769]: pgmap v809: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:42.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:42 smithi130 ceph-mon[32704]: pgmap v809: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:42.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:42 smithi102 ceph-mon[28094]: pgmap v809: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:48:44.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:44 smithi112 ceph-mon[32769]: pgmap v810: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:44.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:44 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:48:44.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:44 smithi130 ceph-mon[32704]: pgmap v810: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:44.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:44 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:48:44.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:44 smithi102 ceph-mon[28094]: pgmap v810: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-26T20:48:44.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:44 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:48:46.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:46 smithi112 ceph-mon[32769]: pgmap v811: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:48:46.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:48:46.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:48:46.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:48:46.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:46 smithi130 ceph-mon[32704]: pgmap v811: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:48:46.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:48:46.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:48:46.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:48:46.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:46 smithi102 ceph-mon[28094]: pgmap v811: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:48:46.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:48:46.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:48:46.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:48:48.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:48 smithi112 ceph-mon[32769]: pgmap v812: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:48:48.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:48 smithi130 ceph-mon[32704]: pgmap v812: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:48:48.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:48 smithi102 ceph-mon[28094]: pgmap v812: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:48:50.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:50 smithi112 ceph-mon[32769]: pgmap v813: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:48:50.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:50 smithi130 ceph-mon[32704]: pgmap v813: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:48:50.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:50 smithi102 ceph-mon[28094]: pgmap v813: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:48:52.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:52 smithi112 ceph-mon[32769]: pgmap v814: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:48:52.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:52 smithi130 ceph-mon[32704]: pgmap v814: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:48:52.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:52 smithi102 ceph-mon[28094]: pgmap v814: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:48:54.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:54 smithi112 ceph-mon[32769]: pgmap v815: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-26T20:48:54.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:54 smithi130 ceph-mon[32704]: pgmap v815: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-26T20:48:54.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:54 smithi102 ceph-mon[28094]: pgmap v815: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-26T20:48:56.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:56 smithi112 ceph-mon[32769]: pgmap v816: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:48:56.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:56 smithi130 ceph-mon[32704]: pgmap v816: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:48:56.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:56 smithi102 ceph-mon[28094]: pgmap v816: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:48:58.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:48:58 smithi112 ceph-mon[32769]: pgmap v817: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:58.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:48:58 smithi130 ceph-mon[32704]: pgmap v817: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:48:58.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:48:58 smithi102 ceph-mon[28094]: pgmap v817: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:00.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:00 smithi112 ceph-mon[32769]: pgmap v818: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:00.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:00 smithi130 ceph-mon[32704]: pgmap v818: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:00.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:00 smithi102 ceph-mon[28094]: pgmap v818: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:02.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:02 smithi112 ceph-mon[32769]: pgmap v819: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:02.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:02 smithi130 ceph-mon[32704]: pgmap v819: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:02.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:02 smithi102 ceph-mon[28094]: pgmap v819: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:04.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:04 smithi112 ceph-mon[32769]: pgmap v820: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:04.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:04 smithi130 ceph-mon[32704]: pgmap v820: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:04.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:04 smithi102 ceph-mon[28094]: pgmap v820: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:06.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:06 smithi112 ceph-mon[32769]: pgmap v821: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:06.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:06 smithi130 ceph-mon[32704]: pgmap v821: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:06.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:06 smithi102 ceph-mon[28094]: pgmap v821: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:08.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:08 smithi112 ceph-mon[32769]: pgmap v822: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:08.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:08 smithi130 ceph-mon[32704]: pgmap v822: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:08.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:08 smithi102 ceph-mon[28094]: pgmap v822: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:10.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:10 smithi130 ceph-mon[32704]: pgmap v823: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:10.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:10 smithi102 ceph-mon[28094]: pgmap v823: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:10.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:10 smithi112 ceph-mon[32769]: pgmap v823: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:12.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:12 smithi130 ceph-mon[32704]: pgmap v824: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:12.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:12 smithi102 ceph-mon[28094]: pgmap v824: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:12.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:12 smithi112 ceph-mon[32769]: pgmap v824: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:14.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:14 smithi130 ceph-mon[32704]: pgmap v825: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:14.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:14 smithi102 ceph-mon[28094]: pgmap v825: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:14.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:14 smithi112 ceph-mon[32769]: pgmap v825: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:16.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:16 smithi130 ceph-mon[32704]: pgmap v826: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:16.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:16 smithi102 ceph-mon[28094]: pgmap v826: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:16.778 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:16 smithi112 ceph-mon[32769]: pgmap v826: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:18.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:18 smithi130 ceph-mon[32704]: pgmap v827: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:18.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:18 smithi102 ceph-mon[28094]: pgmap v827: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:18.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:18 smithi112 ceph-mon[32769]: pgmap v827: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:20.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:20 smithi130 ceph-mon[32704]: pgmap v828: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:20.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:20 smithi102 ceph-mon[28094]: pgmap v828: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:20.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:20 smithi112 ceph-mon[32769]: pgmap v828: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:22.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:22 smithi130 ceph-mon[32704]: pgmap v829: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:22.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:22 smithi102 ceph-mon[28094]: pgmap v829: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:22.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:22 smithi112 ceph-mon[32769]: pgmap v829: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:24.579 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:24 smithi112 ceph-mon[32769]: pgmap v830: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:24.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:24 smithi130 ceph-mon[32704]: pgmap v830: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:24.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:24 smithi102 ceph-mon[28094]: pgmap v830: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:26.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:26 smithi130 ceph-mon[32704]: pgmap v831: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:26.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:26 smithi102 ceph-mon[28094]: pgmap v831: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:26.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:26 smithi112 ceph-mon[32769]: pgmap v831: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:28.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:28 smithi130 ceph-mon[32704]: pgmap v832: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:28.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:28 smithi102 ceph-mon[28094]: pgmap v832: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:28.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:28 smithi112 ceph-mon[32769]: pgmap v832: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:30.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:30 smithi130 ceph-mon[32704]: pgmap v833: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:49:30.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:30 smithi102 ceph-mon[28094]: pgmap v833: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:49:30.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:30 smithi112 ceph-mon[32769]: pgmap v833: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:49:32.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:32 smithi130 ceph-mon[32704]: pgmap v834: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:49:32.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:32 smithi102 ceph-mon[28094]: pgmap v834: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:49:32.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:32 smithi112 ceph-mon[32769]: pgmap v834: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-26T20:49:34.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:34 smithi130 ceph-mon[32704]: pgmap v835: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:49:34.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:34 smithi102 ceph-mon[28094]: pgmap v835: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:49:34.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:34 smithi112 ceph-mon[32769]: pgmap v835: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-26T20:49:36.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:36 smithi130 ceph-mon[32704]: pgmap v836: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-26T20:49:36.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:36 smithi102 ceph-mon[28094]: pgmap v836: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-26T20:49:36.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:36 smithi112 ceph-mon[32769]: pgmap v836: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-04-26T20:49:38.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:38 smithi130 ceph-mon[32704]: pgmap v837: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:49:38.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:38 smithi102 ceph-mon[28094]: pgmap v837: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:49:38.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:38 smithi112 ceph-mon[32769]: pgmap v837: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-26T20:49:40.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:40 smithi130 ceph-mon[32704]: pgmap v838: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:49:40.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:40 smithi102 ceph-mon[28094]: pgmap v838: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:49:40.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:40 smithi112 ceph-mon[32769]: pgmap v838: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-26T20:49:42.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:42 smithi130 ceph-mon[32704]: pgmap v839: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:42.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:42 smithi102 ceph-mon[28094]: pgmap v839: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:42.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:42 smithi112 ceph-mon[32769]: pgmap v839: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:44.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:44 smithi130 ceph-mon[32704]: pgmap v840: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:44.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:44 smithi102 ceph-mon[28094]: pgmap v840: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:44.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:44 smithi112 ceph-mon[32769]: pgmap v840: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:46.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:46 smithi102 ceph-mon[28094]: pgmap v841: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:46.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:46 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:49:46.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:46 smithi112 ceph-mon[32769]: pgmap v841: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:46.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:46 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:49:46.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:46 smithi130 ceph-mon[32704]: pgmap v841: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:46.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:46 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T20:49:47.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:49:47.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:49:47.649 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:47 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:49:47.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:49:47.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:49:47.777 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:47 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:49:47.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T20:49:47.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T20:49:47.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:47 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' 2024-04-26T20:49:49.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:48 smithi102 ceph-mon[28094]: pgmap v842: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:49.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:48 smithi112 ceph-mon[32769]: pgmap v842: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:49.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:48 smithi130 ceph-mon[32704]: pgmap v842: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:51.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:50 smithi102 ceph-mon[28094]: pgmap v843: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:51.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:50 smithi112 ceph-mon[32769]: pgmap v843: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:51.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:50 smithi130 ceph-mon[32704]: pgmap v843: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:49:53.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:52 smithi102 ceph-mon[28094]: pgmap v844: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:53.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:52 smithi112 ceph-mon[32769]: pgmap v844: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:53.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:52 smithi130 ceph-mon[32704]: pgmap v844: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:49:55.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:54 smithi102 ceph-mon[28094]: pgmap v845: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:55.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:54 smithi112 ceph-mon[32769]: pgmap v845: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:55.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:54 smithi130 ceph-mon[32704]: pgmap v845: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:49:56.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:55 smithi112 ceph-mon[32769]: pgmap v846: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:56.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:55 smithi130 ceph-mon[32704]: pgmap v846: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:56.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:55 smithi102 ceph-mon[28094]: pgmap v846: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:49:58.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:49:58 smithi130 ceph-mon[32704]: pgmap v847: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:58.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:49:58 smithi102 ceph-mon[28094]: pgmap v847: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:49:58.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:49:58 smithi112 ceph-mon[32769]: pgmap v847: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:50:00.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:00 smithi102 ceph-mon[28094]: pgmap v848: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:50:00.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:00 smithi102 ceph-mon[28094]: overall HEALTH_OK 2024-04-26T20:50:00.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:00 smithi112 ceph-mon[32769]: pgmap v848: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:50:00.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:00 smithi112 ceph-mon[32769]: overall HEALTH_OK 2024-04-26T20:50:00.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:00 smithi130 ceph-mon[32704]: pgmap v848: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:50:00.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:00 smithi130 ceph-mon[32704]: overall HEALTH_OK 2024-04-26T20:50:02.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:02 smithi102 ceph-mon[28094]: pgmap v849: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:50:02.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:02 smithi112 ceph-mon[32769]: pgmap v849: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:50:02.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:02 smithi130 ceph-mon[32704]: pgmap v849: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:50:04.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:04 smithi112 ceph-mon[32769]: pgmap v850: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:50:04.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:04 smithi130 ceph-mon[32704]: pgmap v850: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:50:04.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:04 smithi102 ceph-mon[28094]: pgmap v850: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:50:06.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:06 smithi112 ceph-mon[32769]: pgmap v851: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:50:06.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:06 smithi130 ceph-mon[32704]: pgmap v851: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:50:06.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:06 smithi102 ceph-mon[28094]: pgmap v851: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:50:08.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:08 smithi112 ceph-mon[32769]: pgmap v852: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:50:08.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:08 smithi130 ceph-mon[32704]: pgmap v852: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:50:08.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:08 smithi102 ceph-mon[28094]: pgmap v852: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:50:10.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:10 smithi112 ceph-mon[32769]: pgmap v853: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:50:10.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:10 smithi130 ceph-mon[32704]: pgmap v853: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:50:10.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:10 smithi102 ceph-mon[28094]: pgmap v853: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:50:12.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:12 smithi112 ceph-mon[32769]: pgmap v854: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:50:12.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:12 smithi130 ceph-mon[32704]: pgmap v854: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:50:12.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:12 smithi102 ceph-mon[28094]: pgmap v854: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:50:14.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:14 smithi112 ceph-mon[32769]: pgmap v855: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:50:14.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:14 smithi130 ceph-mon[32704]: pgmap v855: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:50:14.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:14 smithi102 ceph-mon[28094]: pgmap v855: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-26T20:50:16.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:16 smithi112 ceph-mon[32769]: pgmap v856: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:50:16.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:16 smithi130 ceph-mon[32704]: pgmap v856: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:50:16.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:16 smithi102 ceph-mon[28094]: pgmap v856: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-26T20:50:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:18.359+0000 7f8069800640 1 -- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f804c0a3c30 msgr2=0x7f804c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900021625) for more than 900000000 us, fault. 2024-04-26T20:50:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:18.359+0000 7f8069800640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f804c0a3c30 0x7f804c0a60f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8058005ab0 tx=0x7f8058005a20 comp rx=0 tx=0).stop 2024-04-26T20:50:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:18.359+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f804c0a3c30 msgr2=0x7f804c0a60f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T20:50:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:18.359+0000 7f8062000640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f804c0a3c30 0x7f804c0a60f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:18.359+0000 7f8062000640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f804c0af360 0x7f804c0b1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:18.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:18.359+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f805001ee50 con 0x7f804c0af360 2024-04-26T20:50:18.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:18.360+0000 7f8069800640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f804c0af360 0x7f804c0b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:18.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:18.362+0000 7f8069800640 1 --2- 172.21.15.102:0/806585123 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f804c0af360 0x7f804c0b1750 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8058006010 tx=0x7f8058041000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:18.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:18.478+0000 7f8060c00640 1 -- 172.21.15.102:0/806585123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f80400063d0 con 0x7f80641028c0 2024-04-26T20:50:18.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:18.479+0000 7f8062000640 1 -- 172.21.15.102:0/806585123 <== mon.1 v2:172.21.15.130:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f80400063d0 con 0x7f80641028c0 2024-04-26T20:50:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-04-26T20:50:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-26T20:50:18.616 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-26T20:50:18.920 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=aa14764582a04792bb7c86b55472e911 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-26T20:50:18.924 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-26T20:50:18.936 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T20:50:18.936 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:50:18.936 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T20:50:18.941 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:18 smithi130 ceph-mon[32704]: pgmap v857: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:50:18.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:18 smithi102 ceph-mon[28094]: pgmap v857: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:50:19.001 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-26T20:50:19.001 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T20:50:19.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:18 smithi112 ceph-mon[32769]: pgmap v857: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-26T20:50:19.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:50:19.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.299+0000 7fa2c1800640 1 -- 172.21.15.102:0/1074662598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc105b80 msgr2=0x7fa2bc106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.299+0000 7fa2c1800640 1 --2- 172.21.15.102:0/1074662598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc105b80 0x7fa2bc106000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4009900 tx=0x7fa2a402a410 comp rx=0 tx=0).stop 2024-04-26T20:50:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.299+0000 7fa2c1800640 1 -- 172.21.15.102:0/1074662598 shutdown_connections 2024-04-26T20:50:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.299+0000 7fa2c1800640 1 --2- 172.21.15.102:0/1074662598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc105b80 0x7fa2bc106000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.299+0000 7fa2c1800640 1 --2- 172.21.15.102:0/1074662598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2bc106540 0x7fa2bc10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.299+0000 7fa2c1800640 1 --2- 172.21.15.102:0/1074662598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2bc104990 0x7fa2bc104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.299+0000 7fa2c1800640 1 -- 172.21.15.102:0/1074662598 >> 172.21.15.102:0/1074662598 conn(0x7fa2bc0fff00 msgr2=0x7fa2bc102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:19.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.299+0000 7fa2c1800640 1 -- 172.21.15.102:0/1074662598 shutdown_connections 2024-04-26T20:50:19.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.299+0000 7fa2c1800640 1 -- 172.21.15.102:0/1074662598 wait complete. 2024-04-26T20:50:19.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.300+0000 7fa2c1800640 1 Processor -- start 2024-04-26T20:50:19.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.300+0000 7fa2c1800640 1 -- start start 2024-04-26T20:50:19.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2c1800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2bc104990 0x7fa2bc19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:19.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2c1800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc105b80 0x7fa2bc19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:19.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2c1800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2bc106540 0x7fa2bc1a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2c1800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc110510 con 0x7fa2bc105b80 2024-04-26T20:50:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2c1800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc110390 con 0x7fa2bc104990 2024-04-26T20:50:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2c1800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc110690 con 0x7fa2bc106540 2024-04-26T20:50:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2b3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc105b80 0x7fa2bc19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2b3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc105b80 0x7fa2bc19b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48136/0 (socket says 172.21.15.102:48136) 2024-04-26T20:50:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2b3400640 1 -- 172.21.15.102:0/1204030703 learned_addr learned my addr 172.21.15.102:0/1204030703 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2c0e00640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2bc106540 0x7fa2bc1a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:19.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2bbe00640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2bc104990 0x7fa2bc19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2b3400640 1 -- 172.21.15.102:0/1204030703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2bc106540 msgr2=0x7fa2bc1a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2b3400640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2bc106540 0x7fa2bc1a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2b3400640 1 -- 172.21.15.102:0/1204030703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2bc104990 msgr2=0x7fa2bc19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2b3400640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2bc104990 0x7fa2bc19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2b3400640 1 -- 172.21.15.102:0/1204030703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2bc1a2780 con 0x7fa2bc105b80 2024-04-26T20:50:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.301+0000 7fa2c0e00640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2bc106540 0x7fa2bc1a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:50:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.302+0000 7fa2b3400640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc105b80 0x7fa2bc19b0a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4005ab0 tx=0x7fa2a4003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:19.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.302+0000 7fa2b9600640 1 -- 172.21.15.102:0/1204030703 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a403f070 con 0x7fa2bc105b80 2024-04-26T20:50:19.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.302+0000 7fa2b9600640 1 -- 172.21.15.102:0/1204030703 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2a4031c50 con 0x7fa2bc105b80 2024-04-26T20:50:19.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.302+0000 7fa2b9600640 1 -- 172.21.15.102:0/1204030703 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4035030 con 0x7fa2bc105b80 2024-04-26T20:50:19.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.302+0000 7fa2c1800640 1 -- 172.21.15.102:0/1204030703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2bc1a2a10 con 0x7fa2bc105b80 2024-04-26T20:50:19.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.302+0000 7fa2c1800640 1 -- 172.21.15.102:0/1204030703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2bc1a2eb0 con 0x7fa2bc105b80 2024-04-26T20:50:19.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.303+0000 7fa2c1800640 1 -- 172.21.15.102:0/1204030703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2bc10bcb0 con 0x7fa2bc105b80 2024-04-26T20:50:19.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.305+0000 7fa2b9600640 1 -- 172.21.15.102:0/1204030703 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa2a4042070 con 0x7fa2bc105b80 2024-04-26T20:50:19.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.307+0000 7fa2b9600640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa2a00a3cc0 0x7fa2a00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:19.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.307+0000 7fa2b9600640 1 -- 172.21.15.102:0/1204030703 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fa2a40f0fe0 con 0x7fa2bc105b80 2024-04-26T20:50:19.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.307+0000 7fa2bbe00640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa2a00a3cc0 0x7fa2a00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:19.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.308+0000 7fa2bbe00640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa2a00a3cc0 0x7fa2a00a6180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac0095f0 tx=0x7fa2ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:19.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.312+0000 7fa2b9600640 1 -- 172.21.15.102:0/1204030703 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa2a40bce40 con 0x7fa2bc105b80 2024-04-26T20:50:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.516+0000 7fa2c1800640 1 -- 172.21.15.102:0/1204030703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2bc063530 con 0x7fa2bc105b80 2024-04-26T20:50:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.518+0000 7fa2b9600640 1 -- 172.21.15.102:0/1204030703 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24833 (secure 0 0 0) 0x7fa2a40c1140 con 0x7fa2bc105b80 2024-04-26T20:50:19.518 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:50:19.519 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:34:30.697401+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:50:19.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.520+0000 7fa2c1800640 1 -- 172.21.15.102:0/1204030703 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa2a00a3cc0 msgr2=0x7fa2a00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.520+0000 7fa2c1800640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa2a00a3cc0 0x7fa2a00a6180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac0095f0 tx=0x7fa2ac005f90 comp rx=0 tx=0).stop 2024-04-26T20:50:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.520+0000 7fa2c1800640 1 -- 172.21.15.102:0/1204030703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc105b80 msgr2=0x7fa2bc19b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.520+0000 7fa2c1800640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc105b80 0x7fa2bc19b0a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4005ab0 tx=0x7fa2a4003b90 comp rx=0 tx=0).stop 2024-04-26T20:50:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.520+0000 7fa2c1800640 1 -- 172.21.15.102:0/1204030703 shutdown_connections 2024-04-26T20:50:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.520+0000 7fa2c1800640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fa2a00a3cc0 0x7fa2a00a6180 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.520+0000 7fa2c1800640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc105b80 0x7fa2bc19b0a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.520+0000 7fa2c1800640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2bc106540 0x7fa2bc1a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.520+0000 7fa2c1800640 1 --2- 172.21.15.102:0/1204030703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa2bc104990 0x7fa2bc19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.521+0000 7fa2c1800640 1 -- 172.21.15.102:0/1204030703 >> 172.21.15.102:0/1204030703 conn(0x7fa2bc0fff00 msgr2=0x7fa2bc108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:19.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.521+0000 7fa2c1800640 1 -- 172.21.15.102:0/1204030703 shutdown_connections 2024-04-26T20:50:19.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.521+0000 7fa2c1800640 1 -- 172.21.15.102:0/1204030703 wait complete. 2024-04-26T20:50:19.543 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T20:27:10.527933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T20:32:23.439237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T20:32:24.702263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T20:32:28.348785+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T20:50:19.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T20:50:19.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.754+0000 7f41bfe00640 1 -- 172.21.15.102:0/4237264279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41c0104a50 msgr2=0x7f41c0104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.754+0000 7f41bfe00640 1 --2- 172.21.15.102:0/4237264279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41c0104a50 0x7f41c0104e70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009900 tx=0x7f41b002a410 comp rx=0 tx=0).stop 2024-04-26T20:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.754+0000 7f41bfe00640 1 -- 172.21.15.102:0/4237264279 shutdown_connections 2024-04-26T20:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.754+0000 7f41bfe00640 1 --2- 172.21.15.102:0/4237264279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41c0106600 0x7f41c010ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.754+0000 7f41bfe00640 1 --2- 172.21.15.102:0/4237264279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41c0105c40 0x7f41c01060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.754+0000 7f41bfe00640 1 --2- 172.21.15.102:0/4237264279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41c0104a50 0x7f41c0104e70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.754+0000 7f41bfe00640 1 -- 172.21.15.102:0/4237264279 >> 172.21.15.102:0/4237264279 conn(0x7f41c00fffe0 msgr2=0x7f41c0102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.754+0000 7f41bfe00640 1 -- 172.21.15.102:0/4237264279 shutdown_connections 2024-04-26T20:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.754+0000 7f41bfe00640 1 -- 172.21.15.102:0/4237264279 wait complete. 2024-04-26T20:50:19.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.755+0000 7f41bfe00640 1 Processor -- start 2024-04-26T20:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.755+0000 7f41bfe00640 1 -- start start 2024-04-26T20:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41bfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41c0104a50 0x7f41c019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41bfe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41c0105c40 0x7f41c019b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41bfe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41c0106600 0x7f41c01a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41bfe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01105d0 con 0x7f41c0104a50 2024-04-26T20:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41bfe00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f41c0110450 con 0x7f41c0105c40 2024-04-26T20:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41bfe00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f41c0110750 con 0x7f41c0106600 2024-04-26T20:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41bea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41c0104a50 0x7f41c019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41be000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41c0105c40 0x7f41c019b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41bea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41c0104a50 0x7f41c019aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48144/0 (socket says 172.21.15.102:48144) 2024-04-26T20:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41bea00640 1 -- 172.21.15.102:0/1437987289 learned_addr learned my addr 172.21.15.102:0/1437987289 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41bf400640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41c0106600 0x7f41c01a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41be000640 1 -- 172.21.15.102:0/1437987289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41c0106600 msgr2=0x7f41c01a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41be000640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41c0106600 0x7f41c01a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41be000640 1 -- 172.21.15.102:0/1437987289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41c0104a50 msgr2=0x7f41c019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41be000640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41c0104a50 0x7f41c019aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.756+0000 7f41be000640 1 -- 172.21.15.102:0/1437987289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c01a28e0 con 0x7f41c0105c40 2024-04-26T20:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.757+0000 7f41be000640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41c0105c40 0x7f41c019b1e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00e9f0 tx=0x7f41ac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.757+0000 7f41b7400640 1 -- 172.21.15.102:0/1437987289 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac00cd80 con 0x7f41c0105c40 2024-04-26T20:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.757+0000 7f41bfe00640 1 -- 172.21.15.102:0/1437987289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c01a2bd0 con 0x7f41c0105c40 2024-04-26T20:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.757+0000 7f41b7400640 1 -- 172.21.15.102:0/1437987289 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41ac0035f0 con 0x7f41c0105c40 2024-04-26T20:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.757+0000 7f41b7400640 1 -- 172.21.15.102:0/1437987289 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac018690 con 0x7f41c0105c40 2024-04-26T20:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.757+0000 7f41bfe00640 1 -- 172.21.15.102:0/1437987289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c01a3170 con 0x7f41c0105c40 2024-04-26T20:50:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.758+0000 7f41bfe00640 1 -- 172.21.15.102:0/1437987289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c010be90 con 0x7f41c0105c40 2024-04-26T20:50:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.761+0000 7f41b7400640 1 -- 172.21.15.102:0/1437987289 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f41ac004020 con 0x7f41c0105c40 2024-04-26T20:50:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.762+0000 7f41b7400640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f41a40a3cc0 0x7f41a40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.763+0000 7f41b7400640 1 -- 172.21.15.102:0/1437987289 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f41ac0d0c60 con 0x7f41c0105c40 2024-04-26T20:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.763+0000 7f41bea00640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f41a40a3cc0 0x7f41a40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.764+0000 7f41bea00640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f41a40a3cc0 0x7f41a40a6180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f41b0006010 tx=0x7f41b0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:19.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.764+0000 7f41b7400640 1 -- 172.21.15.102:0/1437987289 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f41ac09d0d0 con 0x7f41c0105c40 2024-04-26T20:50:19.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:19 smithi130 ceph-mon[32704]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:50:19.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:19 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1204030703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:50:19.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:19 smithi102 ceph-mon[28094]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:50:19.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:19 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1204030703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:50:19.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.966+0000 7f41bfe00640 1 -- 172.21.15.102:0/1437987289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f41c002dd20 con 0x7f41c0105c40 2024-04-26T20:50:19.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.967+0000 7f41b7400640 1 -- 172.21.15.102:0/1437987289 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f41ac09c9d0 con 0x7f41c0105c40 2024-04-26T20:50:19.967 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-26T20:50:19.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 -- 172.21.15.102:0/1437987289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f41a40a3cc0 msgr2=0x7f41a40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:19.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f41a40a3cc0 0x7f41a40a6180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f41b0006010 tx=0x7f41b0005a20 comp rx=0 tx=0).stop 2024-04-26T20:50:19.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 -- 172.21.15.102:0/1437987289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41c0105c40 msgr2=0x7f41c019b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:19.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41c0105c40 0x7f41c019b1e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00e9f0 tx=0x7f41ac00eeb0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 -- 172.21.15.102:0/1437987289 shutdown_connections 2024-04-26T20:50:19.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f41a40a3cc0 0x7f41a40a6180 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41c0106600 0x7f41c01a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41c0105c40 0x7f41c019b1e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 --2- 172.21.15.102:0/1437987289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41c0104a50 0x7f41c019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:19.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 -- 172.21.15.102:0/1437987289 >> 172.21.15.102:0/1437987289 conn(0x7f41c00fffe0 msgr2=0x7f41c0109060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:19.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 -- 172.21.15.102:0/1437987289 shutdown_connections 2024-04-26T20:50:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:19.970+0000 7f41bfe00640 1 -- 172.21.15.102:0/1437987289 wait complete. 2024-04-26T20:50:19.990 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T20:50:20.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:19 smithi112 ceph-mon[32769]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T20:50:20.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:19 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1204030703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:50:20.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.197+0000 7f4791200640 1 -- 172.21.15.102:0/970308040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f478c105d80 msgr2=0x7f478c106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:20.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.197+0000 7f4791200640 1 --2- 172.21.15.102:0/970308040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f478c105d80 0x7f478c106200 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4778009900 tx=0x7f477802a410 comp rx=0 tx=0).stop 2024-04-26T20:50:20.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.197+0000 7f4791200640 1 -- 172.21.15.102:0/970308040 shutdown_connections 2024-04-26T20:50:20.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.197+0000 7f4791200640 1 --2- 172.21.15.102:0/970308040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c106740 0x7f478c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.197+0000 7f4791200640 1 --2- 172.21.15.102:0/970308040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f478c105d80 0x7f478c106200 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.197+0000 7f4791200640 1 --2- 172.21.15.102:0/970308040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f478c104b90 0x7f478c104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.197+0000 7f4791200640 1 -- 172.21.15.102:0/970308040 >> 172.21.15.102:0/970308040 conn(0x7f478c1000e0 msgr2=0x7f478c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:20.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.197+0000 7f4791200640 1 -- 172.21.15.102:0/970308040 shutdown_connections 2024-04-26T20:50:20.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.197+0000 7f4791200640 1 -- 172.21.15.102:0/970308040 wait complete. 2024-04-26T20:50:20.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.198+0000 7f4791200640 1 Processor -- start 2024-04-26T20:50:20.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.198+0000 7f4791200640 1 -- start start 2024-04-26T20:50:20.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4791200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c104b90 0x7f478c19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:20.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4791200640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f478c105d80 0x7f478c19b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:20.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4791200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f478c106740 0x7f478c1a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:20.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4791200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f478c1106c0 con 0x7f478c104b90 2024-04-26T20:50:20.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4791200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f478c110540 con 0x7f478c106740 2024-04-26T20:50:20.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4791200640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f478c110840 con 0x7f478c105d80 2024-04-26T20:50:20.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4789e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c104b90 0x7f478c19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:20.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4789e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c104b90 0x7f478c19ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48168/0 (socket says 172.21.15.102:48168) 2024-04-26T20:50:20.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4789e00640 1 -- 172.21.15.102:0/1367040779 learned_addr learned my addr 172.21.15.102:0/1367040779 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:20.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4789400640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f478c105d80 0x7f478c19b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f478a800640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f478c106740 0x7f478c1a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4789e00640 1 -- 172.21.15.102:0/1367040779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f478c105d80 msgr2=0x7f478c19b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.199+0000 7f4789e00640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f478c105d80 0x7f478c19b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.200+0000 7f4789e00640 1 -- 172.21.15.102:0/1367040779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f478c106740 msgr2=0x7f478c1a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.200+0000 7f4789e00640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f478c106740 0x7f478c1a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.200+0000 7f4789e00640 1 -- 172.21.15.102:0/1367040779 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f478c1a2a00 con 0x7f478c104b90 2024-04-26T20:50:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.200+0000 7f4789400640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f478c105d80 0x7f478c19b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:50:20.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.200+0000 7f4789e00640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c104b90 0x7f478c19ade0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f477c00e9c0 tx=0x7f477c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.200+0000 7f4782a00640 1 -- 172.21.15.102:0/1367040779 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c00cd80 con 0x7f478c104b90 2024-04-26T20:50:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.200+0000 7f4782a00640 1 -- 172.21.15.102:0/1367040779 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f477c0035f0 con 0x7f478c104b90 2024-04-26T20:50:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.200+0000 7f4791200640 1 -- 172.21.15.102:0/1367040779 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f478c1a2cf0 con 0x7f478c104b90 2024-04-26T20:50:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.200+0000 7f4782a00640 1 -- 172.21.15.102:0/1367040779 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477c010690 con 0x7f478c104b90 2024-04-26T20:50:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.200+0000 7f4791200640 1 -- 172.21.15.102:0/1367040779 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f478c1a31c0 con 0x7f478c104b90 2024-04-26T20:50:20.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.201+0000 7f4791200640 1 -- 172.21.15.102:0/1367040779 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f478c069ba0 con 0x7f478c104b90 2024-04-26T20:50:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.203+0000 7f4782a00640 1 -- 172.21.15.102:0/1367040779 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f477c004000 con 0x7f478c104b90 2024-04-26T20:50:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.205+0000 7f4782a00640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f47700a3c40 0x7f47700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.205+0000 7f4782a00640 1 -- 172.21.15.102:0/1367040779 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f477c0d0e20 con 0x7f478c104b90 2024-04-26T20:50:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.205+0000 7f4789400640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f47700a3c40 0x7f47700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:20.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.206+0000 7f4789400640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f47700a3c40 0x7f47700a6100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4778000c00 tx=0x7f47780314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:20.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.209+0000 7f4782a00640 1 -- 172.21.15.102:0/1367040779 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f477c09d2f0 con 0x7f478c104b90 2024-04-26T20:50:20.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.410+0000 7f4791200640 1 -- 172.21.15.102:0/1367040779 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f478c02dba0 con 0x7f478c104b90 2024-04-26T20:50:20.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.411+0000 7f4782a00640 1 -- 172.21.15.102:0/1367040779 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f477c09cbf0 con 0x7f478c104b90 2024-04-26T20:50:20.411 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:50:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 -- 172.21.15.102:0/1367040779 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f47700a3c40 msgr2=0x7f47700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f47700a3c40 0x7f47700a6100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4778000c00 tx=0x7f47780314a0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 -- 172.21.15.102:0/1367040779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c104b90 msgr2=0x7f478c19ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c104b90 0x7f478c19ade0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f477c00e9c0 tx=0x7f477c00ee80 comp rx=0 tx=0).stop 2024-04-26T20:50:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 -- 172.21.15.102:0/1367040779 shutdown_connections 2024-04-26T20:50:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f47700a3c40 0x7f47700a6100 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f478c106740 0x7f478c1a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f478c105d80 0x7f478c19b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 --2- 172.21.15.102:0/1367040779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f478c104b90 0x7f478c19ade0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 -- 172.21.15.102:0/1367040779 >> 172.21.15.102:0/1367040779 conn(0x7f478c1000e0 msgr2=0x7f478c109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 -- 172.21.15.102:0/1367040779 shutdown_connections 2024-04-26T20:50:20.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.414+0000 7f4791200640 1 -- 172.21.15.102:0/1367040779 wait complete. 2024-04-26T20:50:20.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T20:50:20.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.647+0000 7f21d1800640 1 -- 172.21.15.102:0/3399299822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc10a5e0 msgr2=0x7f21cc10ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:20.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.647+0000 7f21d1800640 1 --2- 172.21.15.102:0/3399299822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc10a5e0 0x7f21cc10ca80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009ca0 tx=0x7f21c402a950 comp rx=0 tx=0).stop 2024-04-26T20:50:20.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.648+0000 7f21d1800640 1 -- 172.21.15.102:0/3399299822 shutdown_connections 2024-04-26T20:50:20.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.648+0000 7f21d1800640 1 --2- 172.21.15.102:0/3399299822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc10a5e0 0x7f21cc10ca80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.648+0000 7f21d1800640 1 --2- 172.21.15.102:0/3399299822 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f21cc075330 0x7f21cc075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.648+0000 7f21d1800640 1 --2- 172.21.15.102:0/3399299822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21cc076910 0x7f21cc074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.648+0000 7f21d1800640 1 -- 172.21.15.102:0/3399299822 >> 172.21.15.102:0/3399299822 conn(0x7f21cc0fdcc0 msgr2=0x7f21cc100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:20.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.648+0000 7f21d1800640 1 -- 172.21.15.102:0/3399299822 shutdown_connections 2024-04-26T20:50:20.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.648+0000 7f21d1800640 1 -- 172.21.15.102:0/3399299822 wait complete. 2024-04-26T20:50:20.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.649+0000 7f21d1800640 1 Processor -- start 2024-04-26T20:50:20.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.649+0000 7f21d1800640 1 -- start start 2024-04-26T20:50:20.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.649+0000 7f21d1800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f21cc075330 0x7f21cc19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:20.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.649+0000 7f21d1800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21cc076910 0x7f21cc19b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:20.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21d1800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc10a5e0 0x7f21cc1a2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:20.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21d1800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc1104b0 con 0x7f21cc076910 2024-04-26T20:50:20.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21d1800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc110330 con 0x7f21cc10a5e0 2024-04-26T20:50:20.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21d1800640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc110630 con 0x7f21cc075330 2024-04-26T20:50:20.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21cbe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f21cc075330 0x7f21cc19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:20.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21cbe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f21cc075330 0x7f21cc19aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:45286/0 (socket says 172.21.15.102:45286) 2024-04-26T20:50:20.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21cbe00640 1 -- 172.21.15.102:0/3659992590 learned_addr learned my addr 172.21.15.102:0/3659992590 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:20.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21cb400640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21cc076910 0x7f21cc19b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:20.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21d0e00640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc10a5e0 0x7f21cc1a2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:20.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21cbe00640 1 -- 172.21.15.102:0/3659992590 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc10a5e0 msgr2=0x7f21cc1a2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:20.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21cbe00640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc10a5e0 0x7f21cc1a2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21cbe00640 1 -- 172.21.15.102:0/3659992590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21cc076910 msgr2=0x7f21cc19b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:20.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21cbe00640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21cc076910 0x7f21cc19b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.650+0000 7f21cbe00640 1 -- 172.21.15.102:0/3659992590 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21cc1a26e0 con 0x7f21cc075330 2024-04-26T20:50:20.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.651+0000 7f21cbe00640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f21cc075330 0x7f21cc19aac0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00e9f0 tx=0x7f21bc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:20.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.651+0000 7f21c8c00640 1 -- 172.21.15.102:0/3659992590 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc00cd80 con 0x7f21cc075330 2024-04-26T20:50:20.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.651+0000 7f21c8c00640 1 -- 172.21.15.102:0/3659992590 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21bc0108e0 con 0x7f21cc075330 2024-04-26T20:50:20.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.651+0000 7f21d1800640 1 -- 172.21.15.102:0/3659992590 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21cc1a29d0 con 0x7f21cc075330 2024-04-26T20:50:20.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.651+0000 7f21c8c00640 1 -- 172.21.15.102:0/3659992590 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc018980 con 0x7f21cc075330 2024-04-26T20:50:20.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.651+0000 7f21d1800640 1 -- 172.21.15.102:0/3659992590 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21cc1a2f70 con 0x7f21cc075330 2024-04-26T20:50:20.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.652+0000 7f21d1800640 1 -- 172.21.15.102:0/3659992590 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21cc069ac0 con 0x7f21cc075330 2024-04-26T20:50:20.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.654+0000 7f21c8c00640 1 -- 172.21.15.102:0/3659992590 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f21bc010430 con 0x7f21cc075330 2024-04-26T20:50:20.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.655+0000 7f21c8c00640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f21b00a3cc0 0x7f21b00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:20.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.656+0000 7f21c8c00640 1 -- 172.21.15.102:0/3659992590 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f21bc0d0c60 con 0x7f21cc075330 2024-04-26T20:50:20.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.656+0000 7f21cb400640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f21b00a3cc0 0x7f21b00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:20.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.656+0000 7f21cb400640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f21b00a3cc0 0x7f21b00a6180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f21cc19bfd0 tx=0x7f21b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:20.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.660+0000 7f21c8c00640 1 -- 172.21.15.102:0/3659992590 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f21bc09ca80 con 0x7f21cc075330 2024-04-26T20:50:20.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.863+0000 7f21d1800640 1 -- 172.21.15.102:0/3659992590 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f21cc02db90 con 0x7f21cc075330 2024-04-26T20:50:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.864+0000 7f21c8c00640 1 -- 172.21.15.102:0/3659992590 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f21bc0a0d80 con 0x7f21cc075330 2024-04-26T20:50:20.864 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-26T20:50:20.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.867+0000 7f21d1800640 1 -- 172.21.15.102:0/3659992590 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f21b00a3cc0 msgr2=0x7f21b00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:20.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.867+0000 7f21d1800640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f21b00a3cc0 0x7f21b00a6180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f21cc19bfd0 tx=0x7f21b8005f90 comp rx=0 tx=0).stop 2024-04-26T20:50:20.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.867+0000 7f21d1800640 1 -- 172.21.15.102:0/3659992590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f21cc075330 msgr2=0x7f21cc19aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:20.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.867+0000 7f21d1800640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f21cc075330 0x7f21cc19aac0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00e9f0 tx=0x7f21bc00eeb0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.867+0000 7f21d1800640 1 -- 172.21.15.102:0/3659992590 shutdown_connections 2024-04-26T20:50:20.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.867+0000 7f21d1800640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f21b00a3cc0 0x7f21b00a6180 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.867+0000 7f21d1800640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21cc076910 0x7f21cc19b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.867+0000 7f21d1800640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc10a5e0 0x7f21cc1a2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.867+0000 7f21d1800640 1 --2- 172.21.15.102:0/3659992590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f21cc075330 0x7f21cc19aac0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:20.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.867+0000 7f21d1800640 1 -- 172.21.15.102:0/3659992590 >> 172.21.15.102:0/3659992590 conn(0x7f21cc0fdcc0 msgr2=0x7f21cc106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:20.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.868+0000 7f21d1800640 1 -- 172.21.15.102:0/3659992590 shutdown_connections 2024-04-26T20:50:20.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:20.868+0000 7f21d1800640 1 -- 172.21.15.102:0/3659992590 wait complete. 2024-04-26T20:50:20.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:20 smithi130 ceph-mon[32704]: pgmap v858: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:50:20.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:20 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1437987289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:50:20.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:20 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1367040779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:50:20.883 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:20 smithi102 ceph-mon[28094]: pgmap v858: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:50:20.883 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:20 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1437987289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:50:20.884 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:20 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1367040779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:50:20.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T20:50:21.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:20 smithi112 ceph-mon[32769]: pgmap v858: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-26T20:50:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:20 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1437987289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T20:50:21.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:20 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1367040779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T20:50:21.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.097+0000 7f82b8000640 1 -- 172.21.15.102:0/1606081102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b010a180 msgr2=0x7f82b010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.097+0000 7f82b8000640 1 --2- 172.21.15.102:0/1606081102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b010a180 0x7f82b010a600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f829c009900 tx=0x7f829c02a410 comp rx=0 tx=0).stop 2024-04-26T20:50:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.097+0000 7f82b8000640 1 -- 172.21.15.102:0/1606081102 shutdown_connections 2024-04-26T20:50:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.097+0000 7f82b8000640 1 --2- 172.21.15.102:0/1606081102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b010a180 0x7f82b010a600 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.097+0000 7f82b8000640 1 --2- 172.21.15.102:0/1606081102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82b010ab40 0x7f82b01113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.097+0000 7f82b8000640 1 --2- 172.21.15.102:0/1606081102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82b0108f90 0x7f82b01093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.097+0000 7f82b8000640 1 -- 172.21.15.102:0/1606081102 >> 172.21.15.102:0/1606081102 conn(0x7f82b0077f30 msgr2=0x7f82b007a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.097+0000 7f82b8000640 1 -- 172.21.15.102:0/1606081102 shutdown_connections 2024-04-26T20:50:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.097+0000 7f82b8000640 1 -- 172.21.15.102:0/1606081102 wait complete. 2024-04-26T20:50:21.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.098+0000 7f82b8000640 1 Processor -- start 2024-04-26T20:50:21.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.098+0000 7f82b8000640 1 -- start start 2024-04-26T20:50:21.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b8000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82b0108f90 0x7f82b019f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:21.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b8000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b010a180 0x7f82b019f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:21.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b8000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82b010ab40 0x7f82b01a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:21.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b8000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f82b0114ad0 con 0x7f82b010a180 2024-04-26T20:50:21.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b8000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f82b0114950 con 0x7f82b010ab40 2024-04-26T20:50:21.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b8000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f82b0114c50 con 0x7f82b0108f90 2024-04-26T20:50:21.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b010a180 0x7f82b019f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:21.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b5600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82b0108f90 0x7f82b019f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:21.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b5600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82b0108f90 0x7f82b019f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:45308/0 (socket says 172.21.15.102:45308) 2024-04-26T20:50:21.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b5600640 1 -- 172.21.15.102:0/2226625684 learned_addr learned my addr 172.21.15.102:0/2226625684 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:21.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b6000640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82b010ab40 0x7f82b01a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:21.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b5600640 1 -- 172.21.15.102:0/2226625684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82b010ab40 msgr2=0x7f82b01a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:21.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b5600640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82b010ab40 0x7f82b01a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b5600640 1 -- 172.21.15.102:0/2226625684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b010a180 msgr2=0x7f82b019f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:21.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b5600640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b010a180 0x7f82b019f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.099+0000 7f82b5600640 1 -- 172.21.15.102:0/2226625684 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b01a6de0 con 0x7f82b0108f90 2024-04-26T20:50:21.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.100+0000 7f82b5600640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82b0108f90 0x7f82b019f1c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f82a400d950 tx=0x7f82a400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:21.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.100+0000 7f82ae000640 1 -- 172.21.15.102:0/2226625684 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a4017020 con 0x7f82b0108f90 2024-04-26T20:50:21.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.100+0000 7f82ae000640 1 -- 172.21.15.102:0/2226625684 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82a4009c30 con 0x7f82b0108f90 2024-04-26T20:50:21.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.100+0000 7f82b8000640 1 -- 172.21.15.102:0/2226625684 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b01a70d0 con 0x7f82b0108f90 2024-04-26T20:50:21.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.100+0000 7f82ae000640 1 -- 172.21.15.102:0/2226625684 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a400f600 con 0x7f82b0108f90 2024-04-26T20:50:21.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.100+0000 7f82b8000640 1 -- 172.21.15.102:0/2226625684 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b01a75f0 con 0x7f82b0108f90 2024-04-26T20:50:21.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.101+0000 7f82b8000640 1 -- 172.21.15.102:0/2226625684 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82b0110320 con 0x7f82b0108f90 2024-04-26T20:50:21.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.103+0000 7f82ae000640 1 -- 172.21.15.102:0/2226625684 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f82a401a020 con 0x7f82b0108f90 2024-04-26T20:50:21.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.105+0000 7f82ae000640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82980a3c40 0x7f82980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:21.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.105+0000 7f82ae000640 1 -- 172.21.15.102:0/2226625684 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f82a40d0d10 con 0x7f82b0108f90 2024-04-26T20:50:21.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.105+0000 7f82b4c00640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82980a3c40 0x7f82980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:21.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.105+0000 7f82b4c00640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82980a3c40 0x7f82980a6100 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f829c000c00 tx=0x7f829c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:21.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.106+0000 7f82ae000640 1 -- 172.21.15.102:0/2226625684 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f82a4021020 con 0x7f82b0108f90 2024-04-26T20:50:21.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.309+0000 7f82b8000640 1 -- 172.21.15.102:0/2226625684 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f82b0063530 con 0x7f82b0108f90 2024-04-26T20:50:21.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.310+0000 7f82ae000640 1 -- 172.21.15.102:0/2226625684 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f82a409d180 con 0x7f82b0108f90 2024-04-26T20:50:21.311 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-26T20:50:21.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.312+0000 7f82b8000640 1 -- 172.21.15.102:0/2226625684 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82980a3c40 msgr2=0x7f82980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:21.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.312+0000 7f82b8000640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82980a3c40 0x7f82980a6100 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f829c000c00 tx=0x7f829c0314a0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.312+0000 7f82b8000640 1 -- 172.21.15.102:0/2226625684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82b0108f90 msgr2=0x7f82b019f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:21.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.312+0000 7f82b8000640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82b0108f90 0x7f82b019f1c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f82a400d950 tx=0x7f82a400de10 comp rx=0 tx=0).stop 2024-04-26T20:50:21.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.312+0000 7f82b8000640 1 -- 172.21.15.102:0/2226625684 shutdown_connections 2024-04-26T20:50:21.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.312+0000 7f82b8000640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f82980a3c40 0x7f82980a6100 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.312+0000 7f82b8000640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b010a180 0x7f82b019f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.312+0000 7f82b8000640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82b010ab40 0x7f82b01a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.312+0000 7f82b8000640 1 --2- 172.21.15.102:0/2226625684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82b0108f90 0x7f82b019f1c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.313+0000 7f82b8000640 1 -- 172.21.15.102:0/2226625684 >> 172.21.15.102:0/2226625684 conn(0x7f82b0077f30 msgr2=0x7f82b010d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:21.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.313+0000 7f82b8000640 1 -- 172.21.15.102:0/2226625684 shutdown_connections 2024-04-26T20:50:21.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.313+0000 7f82b8000640 1 -- 172.21.15.102:0/2226625684 wait complete. 2024-04-26T20:50:21.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T20:50:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.535+0000 7fe305a00640 1 -- 172.21.15.102:0/3461826769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe300106540 msgr2=0x7fe30010cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.535+0000 7fe305a00640 1 --2- 172.21.15.102:0/3461826769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe300106540 0x7fe30010cdd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009be0 tx=0x7fe2f802a6c0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.535+0000 7fe305a00640 1 -- 172.21.15.102:0/3461826769 shutdown_connections 2024-04-26T20:50:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.535+0000 7fe305a00640 1 --2- 172.21.15.102:0/3461826769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe300106540 0x7fe30010cdd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.535+0000 7fe305a00640 1 --2- 172.21.15.102:0/3461826769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe300105b80 0x7fe300106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.536+0000 7fe305a00640 1 --2- 172.21.15.102:0/3461826769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe300104990 0x7fe300104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.536+0000 7fe305a00640 1 -- 172.21.15.102:0/3461826769 >> 172.21.15.102:0/3461826769 conn(0x7fe3000fff00 msgr2=0x7fe300102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:21.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.536+0000 7fe305a00640 1 -- 172.21.15.102:0/3461826769 shutdown_connections 2024-04-26T20:50:21.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.536+0000 7fe305a00640 1 -- 172.21.15.102:0/3461826769 wait complete. 2024-04-26T20:50:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe305a00640 1 Processor -- start 2024-04-26T20:50:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe305a00640 1 -- start start 2024-04-26T20:50:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe305a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe300104990 0x7fe30019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe305a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe300105b80 0x7fe30019b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe305a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe300106540 0x7fe3001a20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe305a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001104a0 con 0x7fe300106540 2024-04-26T20:50:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe305a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe300110320 con 0x7fe300105b80 2024-04-26T20:50:21.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe305a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe300110620 con 0x7fe300104990 2024-04-26T20:50:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe2ffe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe300104990 0x7fe30019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe305000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe300106540 0x7fe3001a20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe2f7e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe300105b80 0x7fe30019b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe2ffe00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe300104990 0x7fe30019aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:45326/0 (socket says 172.21.15.102:45326) 2024-04-26T20:50:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe2f7e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe300105b80 0x7fe30019b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:53400/0 (socket says 172.21.15.102:53400) 2024-04-26T20:50:21.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.537+0000 7fe2ffe00640 1 -- 172.21.15.102:0/2542844240 learned_addr learned my addr 172.21.15.102:0/2542844240 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.538+0000 7fe2ffe00640 1 -- 172.21.15.102:0/2542844240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe300105b80 msgr2=0x7fe30019b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.538+0000 7fe2ffe00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe300105b80 0x7fe30019b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.538+0000 7fe2ffe00640 1 -- 172.21.15.102:0/2542844240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe300106540 msgr2=0x7fe3001a20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.538+0000 7fe2ffe00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe300106540 0x7fe3001a20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.538+0000 7fe2ffe00640 1 -- 172.21.15.102:0/2542844240 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3001a2710 con 0x7fe300104990 2024-04-26T20:50:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.538+0000 7fe2f7e00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe300105b80 0x7fe30019b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:50:21.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.538+0000 7fe2ffe00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe300104990 0x7fe30019aaf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000e9f0 tx=0x7fe2f000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:21.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.541+0000 7fe2fd600640 1 -- 172.21.15.102:0/2542844240 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f000ccc0 con 0x7fe300104990 2024-04-26T20:50:21.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.541+0000 7fe2fd600640 1 -- 172.21.15.102:0/2542844240 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2f00035f0 con 0x7fe300104990 2024-04-26T20:50:21.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.541+0000 7fe305a00640 1 -- 172.21.15.102:0/2542844240 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3001a2a00 con 0x7fe300104990 2024-04-26T20:50:21.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.541+0000 7fe2fd600640 1 -- 172.21.15.102:0/2542844240 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f000ce60 con 0x7fe300104990 2024-04-26T20:50:21.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.541+0000 7fe305a00640 1 -- 172.21.15.102:0/2542844240 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3001a2fa0 con 0x7fe300104990 2024-04-26T20:50:21.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.543+0000 7fe305a00640 1 -- 172.21.15.102:0/2542844240 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe300069ac0 con 0x7fe300104990 2024-04-26T20:50:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.545+0000 7fe2fd600640 1 -- 172.21.15.102:0/2542844240 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fe2f0004000 con 0x7fe300104990 2024-04-26T20:50:21.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.547+0000 7fe2fd600640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2e40a3cc0 0x7fe2e40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.547+0000 7fe2fd600640 1 -- 172.21.15.102:0/2542844240 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fe2f00d0ad0 con 0x7fe300104990 2024-04-26T20:50:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.547+0000 7fe2f7e00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2e40a3cc0 0x7fe2e40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.548+0000 7fe2f7e00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2e40a3cc0 0x7fe2e40a6180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe30019c000 tx=0x7fe2e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.551+0000 7fe2fd600640 1 -- 172.21.15.102:0/2542844240 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe2f009cfa0 con 0x7fe300104990 2024-04-26T20:50:21.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:21 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3659992590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:50:21.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:21 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2226625684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:50:21.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.768+0000 7fe305a00640 1 -- 172.21.15.102:0/2542844240 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe30010bcb0 con 0x7fe300104990 2024-04-26T20:50:21.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.769+0000 7fe2fd600640 1 -- 172.21.15.102:0/2542844240 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+602 (secure 0 0 0) 0x7fe2f009c8a0 con 0x7fe300104990 2024-04-26T20:50:21.770 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2611652511 2024-04-27T20:24:28.376337+0000 2024-04-26T20:50:21.770 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/1441707916 2024-04-27T20:24:28.376337+0000 2024-04-26T20:50:21.771 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/1441707916 2024-04-27T20:24:28.376337+0000 2024-04-26T20:50:21.771 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2809626135 2024-04-27T20:24:28.376337+0000 2024-04-26T20:50:21.771 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2372720525 2024-04-27T20:23:53.695696+0000 2024-04-26T20:50:21.771 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2628958579 2024-04-27T20:23:53.695696+0000 2024-04-26T20:50:21.771 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/3976810658 2024-04-27T20:23:53.695696+0000 2024-04-26T20:50:21.771 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/3976810658 2024-04-27T20:23:53.695696+0000 2024-04-26T20:50:21.771 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/3271239188 2024-04-27T20:24:28.376337+0000 2024-04-26T20:50:21.771 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1384283564 2024-04-27T20:23:53.695696+0000 2024-04-26T20:50:21.772 INFO:teuthology.orchestra.run.smithi102.stderr:listed 10 entries 2024-04-26T20:50:21.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 -- 172.21.15.102:0/2542844240 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2e40a3cc0 msgr2=0x7fe2e40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2e40a3cc0 0x7fe2e40a6180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe30019c000 tx=0x7fe2e8005f90 comp rx=0 tx=0).stop 2024-04-26T20:50:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 -- 172.21.15.102:0/2542844240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe300104990 msgr2=0x7fe30019aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe300104990 0x7fe30019aaf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000e9f0 tx=0x7fe2f000eeb0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 -- 172.21.15.102:0/2542844240 shutdown_connections 2024-04-26T20:50:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fe2e40a3cc0 0x7fe2e40a6180 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe300106540 0x7fe3001a20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe300105b80 0x7fe30019b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 --2- 172.21.15.102:0/2542844240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe300104990 0x7fe30019aaf0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 -- 172.21.15.102:0/2542844240 >> 172.21.15.102:0/2542844240 conn(0x7fe3000fff00 msgr2=0x7fe300108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 -- 172.21.15.102:0/2542844240 shutdown_connections 2024-04-26T20:50:21.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:21.772+0000 7fe305a00640 1 -- 172.21.15.102:0/2542844240 wait complete. 2024-04-26T20:50:21.792 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T20:50:21.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:50:21.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T20:50:21.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:21 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3659992590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:50:21.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:21 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2226625684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:50:21.895 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-04-26T20:50:21.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:50:21.895 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T20:50:21.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:50:21.944 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-04-26T20:50:22.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:21 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3659992590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T20:50:22.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:21 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2226625684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T20:50:22.034 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T20:50:22.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:50:22.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T20:50:22.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T20:50:22.123 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-04-26T20:50:22.227 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T20:50:22.239 INFO:tasks.cephadm:Teardown begin 2024-04-26T20:50:22.239 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T20:50:22.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:50:22.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.422+0000 7f66e6600640 1 -- 172.21.15.102:0/158135920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e0103b70 msgr2=0x7f66e0103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:22.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.422+0000 7f66e6600640 1 --2- 172.21.15.102:0/158135920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e0103b70 0x7f66e0103ff0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009900 tx=0x7f66c802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.422+0000 7f66e6600640 1 -- 172.21.15.102:0/158135920 shutdown_connections 2024-04-26T20:50:22.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.422+0000 7f66e6600640 1 --2- 172.21.15.102:0/158135920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e0103b70 0x7f66e0103ff0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.422+0000 7f66e6600640 1 --2- 172.21.15.102:0/158135920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e0104530 0x7f66e010adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.422+0000 7f66e6600640 1 --2- 172.21.15.102:0/158135920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0102980 0x7f66e0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.422+0000 7f66e6600640 1 -- 172.21.15.102:0/158135920 >> 172.21.15.102:0/158135920 conn(0x7f66e00fdf10 msgr2=0x7f66e0100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:22.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.422+0000 7f66e6600640 1 -- 172.21.15.102:0/158135920 shutdown_connections 2024-04-26T20:50:22.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.422+0000 7f66e6600640 1 -- 172.21.15.102:0/158135920 wait complete. 2024-04-26T20:50:22.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.423+0000 7f66e6600640 1 Processor -- start 2024-04-26T20:50:22.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66e6600640 1 -- start start 2024-04-26T20:50:22.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66e6600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0102980 0x7f66e00766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:22.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66e6600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e0103b70 0x7f66e0074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:22.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66e6600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e0104530 0x7f66e0075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:22.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66e6600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f66e010e490 con 0x7f66e0103b70 2024-04-26T20:50:22.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66e6600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f66e010e310 con 0x7f66e0102980 2024-04-26T20:50:22.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66e6600640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f66e010e610 con 0x7f66e0104530 2024-04-26T20:50:22.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66df400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0102980 0x7f66e00766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:22.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66df400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0102980 0x7f66e00766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:53410/0 (socket says 172.21.15.102:53410) 2024-04-26T20:50:22.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66df400640 1 -- 172.21.15.102:0/8979476 learned_addr learned my addr 172.21.15.102:0/8979476 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:22.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66dfe00640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e0104530 0x7f66e0075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:22.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.424+0000 7f66dea00640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e0103b70 0x7f66e0074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:22.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66df400640 1 -- 172.21.15.102:0/8979476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e0104530 msgr2=0x7f66e0075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:22.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66df400640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e0104530 0x7f66e0075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66df400640 1 -- 172.21.15.102:0/8979476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e0103b70 msgr2=0x7f66e0074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:22.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66df400640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e0103b70 0x7f66e0074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66df400640 1 -- 172.21.15.102:0/8979476 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e0075af0 con 0x7f66e0102980 2024-04-26T20:50:22.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66dfe00640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e0104530 0x7f66e0075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:50:22.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66df400640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0102980 0x7f66e00766e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f66d000e9c0 tx=0x7f66d000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:22.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66d7e00640 1 -- 172.21.15.102:0/8979476 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d000cd80 con 0x7f66e0102980 2024-04-26T20:50:22.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66d7e00640 1 -- 172.21.15.102:0/8979476 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66d00035f0 con 0x7f66e0102980 2024-04-26T20:50:22.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66d7e00640 1 -- 172.21.15.102:0/8979476 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d0010690 con 0x7f66e0102980 2024-04-26T20:50:22.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.425+0000 7f66e6600640 1 -- 172.21.15.102:0/8979476 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e01a2d00 con 0x7f66e0102980 2024-04-26T20:50:22.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.426+0000 7f66e6600640 1 -- 172.21.15.102:0/8979476 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e01a31d0 con 0x7f66e0102980 2024-04-26T20:50:22.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.427+0000 7f66e6600640 1 -- 172.21.15.102:0/8979476 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66e0068f60 con 0x7f66e0102980 2024-04-26T20:50:22.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.429+0000 7f66d7e00640 1 -- 172.21.15.102:0/8979476 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f66d00037a0 con 0x7f66e0102980 2024-04-26T20:50:22.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.430+0000 7f66d7e00640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66c00a3cc0 0x7f66c00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:22.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.430+0000 7f66d7e00640 1 -- 172.21.15.102:0/8979476 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f66d00d0cc0 con 0x7f66e0102980 2024-04-26T20:50:22.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.430+0000 7f66dea00640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66c00a3cc0 0x7f66c00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:22.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.431+0000 7f66dea00640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66c00a3cc0 0x7f66c00a6180 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f66c8000c00 tx=0x7f66c80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:22.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.435+0000 7f66d7e00640 1 -- 172.21.15.102:0/8979476 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f66e0068f60 con 0x7f66e0102980 2024-04-26T20:50:22.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.685+0000 7f66e6600640 1 -- 172.21.15.102:0/8979476 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66e002dbf0 con 0x7f66e0102980 2024-04-26T20:50:22.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.687+0000 7f66d7e00640 1 -- 172.21.15.102:0/8979476 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11954 (secure 0 0 0) 0x7f66d00d7080 con 0x7f66e0102980 2024-04-26T20:50:22.688 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:50:22.688 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:32:25.769708+0000","modified":"2024-04-26T20:35:17.237868+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24593,"mds_1":24565,"mds_2":24547,"mds_3":24533,"mds_4":24526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24526,24533,24547,24565,24593]},"id":1}]} 2024-04-26T20:50:22.689 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 28 2024-04-26T20:50:22.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 -- 172.21.15.102:0/8979476 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66c00a3cc0 msgr2=0x7f66c00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:22.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66c00a3cc0 0x7f66c00a6180 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f66c8000c00 tx=0x7f66c80314a0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 -- 172.21.15.102:0/8979476 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0102980 msgr2=0x7f66e00766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:22.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0102980 0x7f66e00766e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f66d000e9c0 tx=0x7f66d000ee80 comp rx=0 tx=0).stop 2024-04-26T20:50:22.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 -- 172.21.15.102:0/8979476 shutdown_connections 2024-04-26T20:50:22.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f66c00a3cc0 0x7f66c00a6180 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e0103b70 0x7f66e0074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e0104530 0x7f66e0075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 --2- 172.21.15.102:0/8979476 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e0102980 0x7f66e00766e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 -- 172.21.15.102:0/8979476 >> 172.21.15.102:0/8979476 conn(0x7f66e00fdf10 msgr2=0x7f66e0106f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:22.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 -- 172.21.15.102:0/8979476 shutdown_connections 2024-04-26T20:50:22.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.690+0000 7f66e6600640 1 -- 172.21.15.102:0/8979476 wait complete. 2024-04-26T20:50:22.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:22 smithi102 ceph-mon[28094]: pgmap v859: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:50:22.708 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:22 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2542844240' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:50:22.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T20:50:22.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:22 smithi130 ceph-mon[32704]: pgmap v859: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:50:22.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:22 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2542844240' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:50:22.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.912+0000 7fc9f2400640 1 -- 172.21.15.102:0/2815505568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ec105d60 msgr2=0x7fc9ec1061e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:22.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.912+0000 7fc9f2400640 1 --2- 172.21.15.102:0/2815505568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ec105d60 0x7fc9ec1061e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009900 tx=0x7fc9d802a3f0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.912+0000 7fc9f2400640 1 -- 172.21.15.102:0/2815505568 shutdown_connections 2024-04-26T20:50:22.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.912+0000 7fc9f2400640 1 --2- 172.21.15.102:0/2815505568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ec106720 0x7fc9ec10cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.912+0000 7fc9f2400640 1 --2- 172.21.15.102:0/2815505568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ec105d60 0x7fc9ec1061e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.912+0000 7fc9f2400640 1 --2- 172.21.15.102:0/2815505568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9ec104b70 0x7fc9ec104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.912+0000 7fc9f2400640 1 -- 172.21.15.102:0/2815505568 >> 172.21.15.102:0/2815505568 conn(0x7fc9ec100120 msgr2=0x7fc9ec102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:22.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.912+0000 7fc9f2400640 1 -- 172.21.15.102:0/2815505568 shutdown_connections 2024-04-26T20:50:22.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.912+0000 7fc9f2400640 1 -- 172.21.15.102:0/2815505568 wait complete. 2024-04-26T20:50:22.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.913+0000 7fc9f2400640 1 Processor -- start 2024-04-26T20:50:22.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.913+0000 7fc9f2400640 1 -- start start 2024-04-26T20:50:22.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9f2400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ec104b70 0x7fc9ec19ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:22.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9f2400640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ec105d60 0x7fc9ec19b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:22.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9f2400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:22.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9f2400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec1106f0 con 0x7fc9ec106720 2024-04-26T20:50:22.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9f2400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec110570 con 0x7fc9ec104b70 2024-04-26T20:50:22.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9f2400640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec110870 con 0x7fc9ec105d60 2024-04-26T20:50:22.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:22.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9eb400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ec104b70 0x7fc9ec19ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:22.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48278/0 (socket says 172.21.15.102:48278) 2024-04-26T20:50:22.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9eb400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ec104b70 0x7fc9ec19ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:53418/0 (socket says 172.21.15.102:53418) 2024-04-26T20:50:22.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9ebe00640 1 -- 172.21.15.102:0/1109111318 learned_addr learned my addr 172.21.15.102:0/1109111318 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:22.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9eaa00640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ec105d60 0x7fc9ec19b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:22.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.914+0000 7fc9eb400640 1 -- 172.21.15.102:0/1109111318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ec105d60 msgr2=0x7fc9ec19b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:22.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9eb400640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ec105d60 0x7fc9ec19b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9eb400640 1 -- 172.21.15.102:0/1109111318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9ec106720 msgr2=0x7fc9ec1a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:22.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9eb400640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:22.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9eb400640 1 -- 172.21.15.102:0/1109111318 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ec1a2980 con 0x7fc9ec104b70 2024-04-26T20:50:22.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9ebe00640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:50:22.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9eb400640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ec104b70 0x7fc9ec19ad60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00e9c0 tx=0x7fc9dc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:22.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9e3e00640 1 -- 172.21.15.102:0/1109111318 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc00ccc0 con 0x7fc9ec104b70 2024-04-26T20:50:22.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9f2400640 1 -- 172.21.15.102:0/1109111318 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ec1a2c70 con 0x7fc9ec104b70 2024-04-26T20:50:22.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9e3e00640 1 -- 172.21.15.102:0/1109111318 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9dc0035f0 con 0x7fc9ec104b70 2024-04-26T20:50:22.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9e3e00640 1 -- 172.21.15.102:0/1109111318 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc00ce60 con 0x7fc9ec104b70 2024-04-26T20:50:22.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.915+0000 7fc9f2400640 1 -- 172.21.15.102:0/1109111318 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ec1a3190 con 0x7fc9ec104b70 2024-04-26T20:50:22.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.916+0000 7fc9f2400640 1 -- 172.21.15.102:0/1109111318 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9ec10bee0 con 0x7fc9ec104b70 2024-04-26T20:50:22.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.918+0000 7fc9e3e00640 1 -- 172.21.15.102:0/1109111318 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc9dc004000 con 0x7fc9ec104b70 2024-04-26T20:50:22.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.919+0000 7fc9e3e00640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc9d00a3c40 0x7fc9d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:22.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.920+0000 7fc9e3e00640 1 -- 172.21.15.102:0/1109111318 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fc9dc0d0f30 con 0x7fc9ec104b70 2024-04-26T20:50:22.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.920+0000 7fc9eaa00640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc9d00a3c40 0x7fc9d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:22.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.921+0000 7fc9eaa00640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc9d00a3c40 0x7fc9d00a6100 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8031500 tx=0x7fc9d8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:22.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:22.922+0000 7fc9e3e00640 1 -- 172.21.15.102:0/1109111318 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc9dc09d3a0 con 0x7fc9ec104b70 2024-04-26T20:50:23.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:22 smithi112 ceph-mon[32769]: pgmap v859: 129 pgs: 129 active+clean; 642 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-26T20:50:23.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:22 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2542844240' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T20:50:23.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.163+0000 7fc9f2400640 1 -- 172.21.15.102:0/1109111318 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9ec02dd40 con 0x7fc9ec104b70 2024-04-26T20:50:23.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.164+0000 7fc9e3e00640 1 -- 172.21.15.102:0/1109111318 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11954 (secure 0 0 0) 0x7fc9dc09cca0 con 0x7fc9ec104b70 2024-04-26T20:50:23.165 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:50:23.165 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/3853049115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":3853049115},{"type":"v1","addr":"172.21.15.102:6835","nonce":3853049115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/2992258222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":2992258222},{"type":"v1","addr":"172.21.15.102:6841","nonce":2992258222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T20:32:25.769708+0000","modified":"2024-04-26T20:35:17.237868+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24593,"mds_1":24565,"mds_2":24547,"mds_3":24533,"mds_4":24526},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/4251888549","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":4251888549},{"type":"v1","addr":"172.21.15.130:6833","nonce":4251888549}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/1543441778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":1543441778},{"type":"v1","addr":"172.21.15.112:6833","nonce":1543441778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3283929332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3283929332},{"type":"v1","addr":"172.21.15.102:6837","nonce":3283929332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/2010209065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":2010209065},{"type":"v1","addr":"172.21.15.112:6835","nonce":2010209065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24514":{"gid":24514,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/829567775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":829567775},{"type":"v1","addr":"172.21.15.130:6835","nonce":829567775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6837/2107305806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2107305806},{"type":"v1","addr":"172.21.15.130:6837","nonce":2107305806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.112:6837/82964069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":82964069},{"type":"v1","addr":"172.21.15.112:6837","nonce":82964069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.130:6839/1700008514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":1700008514},{"type":"v1","addr":"172.21.15.130:6839","nonce":1700008514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.102:6839/3473487681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3473487681},{"type":"v1","addr":"172.21.15.102:6839","nonce":3473487681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/2318597180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":2318597180},{"type":"v1","addr":"172.21.15.112:6839","nonce":2318597180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24526,24533,24547,24565,24593]},"id":1}]} 2024-04-26T20:50:23.166 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 28 2024-04-26T20:50:23.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 -- 172.21.15.102:0/1109111318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc9d00a3c40 msgr2=0x7fc9d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:23.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc9d00a3c40 0x7fc9d00a6100 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8031500 tx=0x7fc9d8005a20 comp rx=0 tx=0).stop 2024-04-26T20:50:23.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 -- 172.21.15.102:0/1109111318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ec104b70 msgr2=0x7fc9ec19ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:23.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ec104b70 0x7fc9ec19ad60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00e9c0 tx=0x7fc9dc00ee80 comp rx=0 tx=0).stop 2024-04-26T20:50:23.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 -- 172.21.15.102:0/1109111318 shutdown_connections 2024-04-26T20:50:23.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fc9d00a3c40 0x7fc9d00a6100 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9ec106720 0x7fc9ec1a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ec105d60 0x7fc9ec19b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 --2- 172.21.15.102:0/1109111318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9ec104b70 0x7fc9ec19ad60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 -- 172.21.15.102:0/1109111318 >> 172.21.15.102:0/1109111318 conn(0x7fc9ec100120 msgr2=0x7fc9ec1090b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:23.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 -- 172.21.15.102:0/1109111318 shutdown_connections 2024-04-26T20:50:23.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.167+0000 7fc9f2400640 1 -- 172.21.15.102:0/1109111318 wait complete. 2024-04-26T20:50:23.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T20:50:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.389+0000 7fd907e00640 1 -- 172.21.15.102:0/2722481722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd90010a830 msgr2=0x7fd90010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.389+0000 7fd907e00640 1 --2- 172.21.15.102:0/2722481722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd90010a830 0x7fd90010ccc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009ca0 tx=0x7fd8f802a8c0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.390+0000 7fd907e00640 1 -- 172.21.15.102:0/2722481722 shutdown_connections 2024-04-26T20:50:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.390+0000 7fd907e00640 1 --2- 172.21.15.102:0/2722481722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd90010a830 0x7fd90010ccc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.390+0000 7fd907e00640 1 --2- 172.21.15.102:0/2722481722 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd900075390 0x7fd900075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.390+0000 7fd907e00640 1 --2- 172.21.15.102:0/2722481722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd900076970 0x7fd900074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.390+0000 7fd907e00640 1 -- 172.21.15.102:0/2722481722 >> 172.21.15.102:0/2722481722 conn(0x7fd9000fdee0 msgr2=0x7fd900100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.390+0000 7fd907e00640 1 -- 172.21.15.102:0/2722481722 shutdown_connections 2024-04-26T20:50:23.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.390+0000 7fd907e00640 1 -- 172.21.15.102:0/2722481722 wait complete. 2024-04-26T20:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd907e00640 1 Processor -- start 2024-04-26T20:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd907e00640 1 -- start start 2024-04-26T20:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd907e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd900075390 0x7fd90019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd907e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd900076970 0x7fd90019b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd907e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd90010a830 0x7fd9001a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd907e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd9001106b0 con 0x7fd900076970 2024-04-26T20:50:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd907e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd900110530 con 0x7fd900075390 2024-04-26T20:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd907e00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd900110830 con 0x7fd90010a830 2024-04-26T20:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd904a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd900076970 0x7fd90019b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd904a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd900076970 0x7fd90019b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48288/0 (socket says 172.21.15.102:48288) 2024-04-26T20:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd904a00640 1 -- 172.21.15.102:0/3124766776 learned_addr learned my addr 172.21.15.102:0/3124766776 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd905400640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd900075390 0x7fd90019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.391+0000 7fd905e00640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd90010a830 0x7fd9001a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd905400640 1 -- 172.21.15.102:0/3124766776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd90010a830 msgr2=0x7fd9001a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd905400640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd90010a830 0x7fd9001a22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd905400640 1 -- 172.21.15.102:0/3124766776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd900076970 msgr2=0x7fd90019b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd905400640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd900076970 0x7fd90019b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd905400640 1 -- 172.21.15.102:0/3124766776 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9001a2950 con 0x7fd900075390 2024-04-26T20:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd904a00640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd900076970 0x7fd90019b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd905400640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd900075390 0x7fd90019ad40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400e9f0 tx=0x7fd8f400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd8fe000640 1 -- 172.21.15.102:0/3124766776 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400ccc0 con 0x7fd900075390 2024-04-26T20:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd907e00640 1 -- 172.21.15.102:0/3124766776 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9001a2c40 con 0x7fd900075390 2024-04-26T20:50:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd8fe000640 1 -- 172.21.15.102:0/3124766776 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8f40108e0 con 0x7fd900075390 2024-04-26T20:50:23.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.392+0000 7fd8fe000640 1 -- 172.21.15.102:0/3124766776 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4018880 con 0x7fd900075390 2024-04-26T20:50:23.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.393+0000 7fd907e00640 1 -- 172.21.15.102:0/3124766776 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9001a3190 con 0x7fd900075390 2024-04-26T20:50:23.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.394+0000 7fd907e00640 1 -- 172.21.15.102:0/3124766776 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd900109560 con 0x7fd900075390 2024-04-26T20:50:23.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.396+0000 7fd8fe000640 1 -- 172.21.15.102:0/3124766776 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fd8f4010430 con 0x7fd900075390 2024-04-26T20:50:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.397+0000 7fd8fe000640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd8e80a3c40 0x7fd8e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.397+0000 7fd8fe000640 1 -- 172.21.15.102:0/3124766776 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fd8f40d11b0 con 0x7fd900075390 2024-04-26T20:50:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.398+0000 7fd904a00640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd8e80a3c40 0x7fd8e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.398+0000 7fd904a00640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd8e80a3c40 0x7fd8e80a6100 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec011550 tx=0x7fd8ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:23.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.399+0000 7fd8fe000640 1 -- 172.21.15.102:0/3124766776 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd8f409cfd0 con 0x7fd900075390 2024-04-26T20:50:23.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.609+0000 7fd907e00640 1 -- 172.21.15.102:0/3124766776 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd90002dcc0 con 0x7fd900075390 2024-04-26T20:50:23.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.611+0000 7fd8fe000640 1 -- 172.21.15.102:0/3124766776 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24833 (secure 0 0 0) 0x7fd8f40a12d0 con 0x7fd900075390 2024-04-26T20:50:23.613 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-26T20:50:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"a94c830a-040a-11ef-bc93-c7b262605968","created":"2024-04-26T20:23:20.101386+0000","modified":"2024-04-26T20:34:30.697401+0000","last_up_change":"2024-04-26T20:31:27.941025+0000","last_in_change":"2024-04-26T20:31:07.548913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T20:27:10.527933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T20:32:23.439237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T20:32:24.702263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T20:32:28.348785+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b00e64c1-d8b2-4ede-a6e5-236cac247fd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6803","nonce":1690985624}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6805","nonce":1690985624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6809","nonce":1690985624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1690985624},{"type":"v1","addr":"172.21.15.102:6807","nonce":1690985624}]},"public_addr":"172.21.15.102:6803/1690985624","cluster_addr":"172.21.15.102:6805/1690985624","heartbeat_back_addr":"172.21.15.102:6809/1690985624","heartbeat_front_addr":"172.21.15.102:6807/1690985624","state":["exists","up"]},{"osd":1,"uuid":"e10d054b-2732-48ef-a2ba-4137acda22bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6801","nonce":3112086050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6803","nonce":3112086050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6807","nonce":3112086050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3112086050},{"type":"v1","addr":"172.21.15.112:6805","nonce":3112086050}]},"public_addr":"172.21.15.112:6801/3112086050","cluster_addr":"172.21.15.112:6803/3112086050","heartbeat_back_addr":"172.21.15.112:6807/3112086050","heartbeat_front_addr":"172.21.15.112:6805/3112086050","state":["exists","up"]},{"osd":2,"uuid":"7d090636-b81e-404e-9b1d-d1919ab42ba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6801","nonce":2227083221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6803","nonce":2227083221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6807","nonce":2227083221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2227083221},{"type":"v1","addr":"172.21.15.130:6805","nonce":2227083221}]},"public_addr":"172.21.15.130:6801/2227083221","cluster_addr":"172.21.15.130:6803/2227083221","heartbeat_back_addr":"172.21.15.130:6807/2227083221","heartbeat_front_addr":"172.21.15.130:6805/2227083221","state":["exists","up"]},{"osd":3,"uuid":"a6d4daaf-ac9a-4897-a74a-9327aef16d22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6811","nonce":819862253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6813","nonce":819862253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6817","nonce":819862253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":819862253},{"type":"v1","addr":"172.21.15.102:6815","nonce":819862253}]},"public_addr":"172.21.15.102:6811/819862253","cluster_addr":"172.21.15.102:6813/819862253","heartbeat_back_addr":"172.21.15.102:6817/819862253","heartbeat_front_addr":"172.21.15.102:6815/819862253","state":["exists","up"]},{"osd":4,"uuid":"4fe3b1fe-1d6f-4ce9-b162-fca16bacd628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6809","nonce":926818023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6811","nonce":926818023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6815","nonce":926818023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":926818023},{"type":"v1","addr":"172.21.15.112:6813","nonce":926818023}]},"public_addr":"172.21.15.112:6809/926818023","cluster_addr":"172.21.15.112:6811/926818023","heartbeat_back_addr":"172.21.15.112:6815/926818023","heartbeat_front_addr":"172.21.15.112:6813/926818023","state":["exists","up"]},{"osd":5,"uuid":"7d936c15-c743-4cef-95fc-f394bf75cdf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6809","nonce":1600420377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6811","nonce":1600420377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6815","nonce":1600420377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1600420377},{"type":"v1","addr":"172.21.15.130:6813","nonce":1600420377}]},"public_addr":"172.21.15.130:6809/1600420377","cluster_addr":"172.21.15.130:6811/1600420377","heartbeat_back_addr":"172.21.15.130:6815/1600420377","heartbeat_front_addr":"172.21.15.130:6813/1600420377","state":["exists","up"]},{"osd":6,"uuid":"3bbd0cd9-f5a4-4453-860d-2884ccfa71f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6819","nonce":879532763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6821","nonce":879532763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6825","nonce":879532763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":879532763},{"type":"v1","addr":"172.21.15.102:6823","nonce":879532763}]},"public_addr":"172.21.15.102:6819/879532763","cluster_addr":"172.21.15.102:6821/879532763","heartbeat_back_addr":"172.21.15.102:6825/879532763","heartbeat_front_addr":"172.21.15.102:6823/879532763","state":["exists","up"]},{"osd":7,"uuid":"4ebc098f-21d8-4b10-a1cb-2793d0675c44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6817","nonce":971991599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6819","nonce":971991599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6823","nonce":971991599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":971991599},{"type":"v1","addr":"172.21.15.112:6821","nonce":971991599}]},"public_addr":"172.21.15.112:6817/971991599","cluster_addr":"172.21.15.112:6819/971991599","heartbeat_back_addr":"172.21.15.112:6823/971991599","heartbeat_front_addr":"172.21.15.112:6821/971991599","state":["exists","up"]},{"osd":8,"uuid":"8a61ad14-24c4-4658-938f-93e58b88ad8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6817","nonce":3061190043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6819","nonce":3061190043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6823","nonce":3061190043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":3061190043},{"type":"v1","addr":"172.21.15.130:6821","nonce":3061190043}]},"public_addr":"172.21.15.130:6817/3061190043","cluster_addr":"172.21.15.130:6819/3061190043","heartbeat_back_addr":"172.21.15.130:6823/3061190043","heartbeat_front_addr":"172.21.15.130:6821/3061190043","state":["exists","up"]},{"osd":9,"uuid":"39e80aea-2edc-49c1-8ab8-46d301e58694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6827","nonce":4267695892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6829","nonce":4267695892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6833","nonce":4267695892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":4267695892},{"type":"v1","addr":"172.21.15.102:6831","nonce":4267695892}]},"public_addr":"172.21.15.102:6827/4267695892","cluster_addr":"172.21.15.102:6829/4267695892","heartbeat_back_addr":"172.21.15.102:6833/4267695892","heartbeat_front_addr":"172.21.15.102:6831/4267695892","state":["exists","up"]},{"osd":10,"uuid":"bd9ac886-cff6-421d-8d22-fb1680e44c02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6825","nonce":2204137717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6827","nonce":2204137717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6831","nonce":2204137717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2204137717},{"type":"v1","addr":"172.21.15.112:6829","nonce":2204137717}]},"public_addr":"172.21.15.112:6825/2204137717","cluster_addr":"172.21.15.112:6827/2204137717","heartbeat_back_addr":"172.21.15.112:6831/2204137717","heartbeat_front_addr":"172.21.15.112:6829/2204137717","state":["exists","up"]},{"osd":11,"uuid":"bc84bfd4-0926-4025-bb47-9a6a1e648b32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6825","nonce":822414281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6827","nonce":822414281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6831","nonce":822414281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":822414281},{"type":"v1","addr":"172.21.15.130:6829","nonce":822414281}]},"public_addr":"172.21.15.130:6825/822414281","cluster_addr":"172.21.15.130:6827/822414281","heartbeat_back_addr":"172.21.15.130:6831/822414281","heartbeat_front_addr":"172.21.15.130:6829/822414281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:20.012377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:26:44.887247+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:08.898013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:27:41.198200+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:07.006761+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:28:31.983155+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:05.018267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:32.157925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:29:57.911180+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:32.007265+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:30:59.962702+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T20:31:25.822591+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2611652511":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6801/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:6800/1441707916":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2809626135":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/2372720525":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/2628958579":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6800/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:6801/3976810658":"2024-04-27T20:23:53.695696+0000","172.21.15.102:0/3271239188":"2024-04-27T20:24:28.376337+0000","172.21.15.102:0/1384283564":"2024-04-27T20:23:53.695696+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T20:50:23.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 -- 172.21.15.102:0/3124766776 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd8e80a3c40 msgr2=0x7fd8e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:23.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd8e80a3c40 0x7fd8e80a6100 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec011550 tx=0x7fd8ec005f90 comp rx=0 tx=0).stop 2024-04-26T20:50:23.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 -- 172.21.15.102:0/3124766776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd900075390 msgr2=0x7fd90019ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:23.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd900075390 0x7fd90019ad40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400e9f0 tx=0x7fd8f400eeb0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 -- 172.21.15.102:0/3124766776 shutdown_connections 2024-04-26T20:50:23.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7fd8e80a3c40 0x7fd8e80a6100 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd900076970 0x7fd90019b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd90010a830 0x7fd9001a22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 --2- 172.21.15.102:0/3124766776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd900075390 0x7fd90019ad40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 -- 172.21.15.102:0/3124766776 >> 172.21.15.102:0/3124766776 conn(0x7fd9000fdee0 msgr2=0x7fd900106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:23.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 -- 172.21.15.102:0/3124766776 shutdown_connections 2024-04-26T20:50:23.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.615+0000 7fd907e00640 1 -- 172.21.15.102:0/3124766776 wait complete. 2024-04-26T20:50:23.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T20:50:23.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.841+0000 7ffa21e00640 1 -- 172.21.15.102:0/3758682447 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c078530 msgr2=0x7ffa1c078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:23.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.841+0000 7ffa21e00640 1 --2- 172.21.15.102:0/3758682447 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c078530 0x7ffa1c078990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009900 tx=0x7ffa0402a3f0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.841+0000 7ffa21e00640 1 -- 172.21.15.102:0/3758682447 shutdown_connections 2024-04-26T20:50:23.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.841+0000 7ffa21e00640 1 --2- 172.21.15.102:0/3758682447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa1c078ed0 0x7ffa1c079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.841+0000 7ffa21e00640 1 --2- 172.21.15.102:0/3758682447 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c078530 0x7ffa1c078990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.841+0000 7ffa21e00640 1 --2- 172.21.15.102:0/3758682447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa1c0772f0 0x7ffa1c077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.841+0000 7ffa21e00640 1 -- 172.21.15.102:0/3758682447 >> 172.21.15.102:0/3758682447 conn(0x7ffa1c0fff00 msgr2=0x7ffa1c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.842+0000 7ffa21e00640 1 -- 172.21.15.102:0/3758682447 shutdown_connections 2024-04-26T20:50:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.842+0000 7ffa21e00640 1 -- 172.21.15.102:0/3758682447 wait complete. 2024-04-26T20:50:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa21e00640 1 Processor -- start 2024-04-26T20:50:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa21e00640 1 -- start start 2024-04-26T20:50:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa21e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa1c0772f0 0x7ffa1c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:23.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa21e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c078530 0x7ffa1c19f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:23.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa21e00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa1c078ed0 0x7ffa1c1a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:23.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa21e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c114910 con 0x7ffa1c0772f0 2024-04-26T20:50:23.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa21e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c114790 con 0x7ffa1c078530 2024-04-26T20:50:23.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa21e00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c114a90 con 0x7ffa1c078ed0 2024-04-26T20:50:23.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa13e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c078530 0x7ffa1c19f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:23.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa13e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c078530 0x7ffa1c19f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:53456/0 (socket says 172.21.15.102:53456) 2024-04-26T20:50:23.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa20a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa1c0772f0 0x7ffa1c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.843+0000 7ffa13e00640 1 -- 172.21.15.102:0/2849682965 learned_addr learned my addr 172.21.15.102:0/2849682965 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa21400640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa1c078ed0 0x7ffa1c1a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa13e00640 1 -- 172.21.15.102:0/2849682965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa1c078ed0 msgr2=0x7ffa1c1a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa13e00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa1c078ed0 0x7ffa1c1a6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa13e00640 1 -- 172.21.15.102:0/2849682965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa1c0772f0 msgr2=0x7ffa1c19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa13e00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa1c0772f0 0x7ffa1c19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa13e00640 1 -- 172.21.15.102:0/2849682965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1c1a6bb0 con 0x7ffa1c078530 2024-04-26T20:50:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa13e00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c078530 0x7ffa1c19f4d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ffa04005ab0 tx=0x7ffa04003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:23.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa1a000640 1 -- 172.21.15.102:0/2849682965 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0403f070 con 0x7ffa1c078530 2024-04-26T20:50:23.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa1a000640 1 -- 172.21.15.102:0/2849682965 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa04031c50 con 0x7ffa1c078530 2024-04-26T20:50:23.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa1a000640 1 -- 172.21.15.102:0/2849682965 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa04035030 con 0x7ffa1c078530 2024-04-26T20:50:23.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.844+0000 7ffa21e00640 1 -- 172.21.15.102:0/2849682965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1c1a6e40 con 0x7ffa1c078530 2024-04-26T20:50:23.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.845+0000 7ffa21e00640 1 -- 172.21.15.102:0/2849682965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1c1a72b0 con 0x7ffa1c078530 2024-04-26T20:50:23.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.846+0000 7ffa21e00640 1 -- 172.21.15.102:0/2849682965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa1c111300 con 0x7ffa1c078530 2024-04-26T20:50:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.848+0000 7ffa1a000640 1 -- 172.21.15.102:0/2849682965 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ffa04042070 con 0x7ffa1c078530 2024-04-26T20:50:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.849+0000 7ffa1a000640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ffa000a3cc0 0x7ffa000a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.850+0000 7ffa1a000640 1 -- 172.21.15.102:0/2849682965 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7ffa040f1730 con 0x7ffa1c078530 2024-04-26T20:50:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.850+0000 7ffa20a00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ffa000a3cc0 0x7ffa000a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.850+0000 7ffa20a00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ffa000a3cc0 0x7ffa000a6180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009bd0 tx=0x7ffa0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:23.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:23.851+0000 7ffa1a000640 1 -- 172.21.15.102:0/2849682965 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffa040bd550 con 0x7ffa1c078530 2024-04-26T20:50:23.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:23 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/8979476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:50:23.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:23 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1109111318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:50:23.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:23 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/8979476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:50:23.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:23 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1109111318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:50:24.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:23 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/8979476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:50:24.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:23 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1109111318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T20:50:24.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.094+0000 7ffa21e00640 1 -- 172.21.15.102:0/2849682965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ffa1c0635d0 con 0x7ffa1c078530 2024-04-26T20:50:24.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.105+0000 7ffa1a000640 1 -- 172.21.15.102:0/2849682965 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) ==== 161+0+0 (secure 0 0 0) 0x7ffa040c1850 con 0x7ffa1c078530 2024-04-26T20:50:24.106 INFO:teuthology.orchestra.run.smithi102.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T20:50:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.108+0000 7ffa21e00640 1 -- 172.21.15.102:0/2849682965 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ffa000a3cc0 msgr2=0x7ffa000a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.108+0000 7ffa21e00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ffa000a3cc0 0x7ffa000a6180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009bd0 tx=0x7ffa0c005f90 comp rx=0 tx=0).stop 2024-04-26T20:50:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.108+0000 7ffa21e00640 1 -- 172.21.15.102:0/2849682965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c078530 msgr2=0x7ffa1c19f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.108+0000 7ffa21e00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c078530 0x7ffa1c19f4d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ffa04005ab0 tx=0x7ffa04003b90 comp rx=0 tx=0).stop 2024-04-26T20:50:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.108+0000 7ffa21e00640 1 -- 172.21.15.102:0/2849682965 shutdown_connections 2024-04-26T20:50:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.108+0000 7ffa21e00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ffa000a3cc0 0x7ffa000a6180 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.108+0000 7ffa21e00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa1c078ed0 0x7ffa1c1a6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.108+0000 7ffa21e00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa1c078530 0x7ffa1c19f4d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.108+0000 7ffa21e00640 1 --2- 172.21.15.102:0/2849682965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa1c0772f0 0x7ffa1c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.108+0000 7ffa21e00640 1 -- 172.21.15.102:0/2849682965 >> 172.21.15.102:0/2849682965 conn(0x7ffa1c0fff00 msgr2=0x7ffa1c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.109+0000 7ffa21e00640 1 -- 172.21.15.102:0/2849682965 shutdown_connections 2024-04-26T20:50:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.109+0000 7ffa21e00640 1 -- 172.21.15.102:0/2849682965 wait complete. 2024-04-26T20:50:24.129 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T20:50:24.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.317+0000 7ff093000640 1 -- 172.21.15.102:0/689841575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff08c104b90 msgr2=0x7ff08c104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:24.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.317+0000 7ff093000640 1 --2- 172.21.15.102:0/689841575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff08c104b90 0x7ff08c104fb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009900 tx=0x7ff07c02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:50:24.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.317+0000 7ff093000640 1 -- 172.21.15.102:0/689841575 shutdown_connections 2024-04-26T20:50:24.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.317+0000 7ff093000640 1 --2- 172.21.15.102:0/689841575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff08c105d80 0x7ff08c106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:24.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.317+0000 7ff093000640 1 --2- 172.21.15.102:0/689841575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c106740 0x7ff08c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:24.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.317+0000 7ff093000640 1 --2- 172.21.15.102:0/689841575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff08c104b90 0x7ff08c104fb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:24.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.317+0000 7ff093000640 1 -- 172.21.15.102:0/689841575 >> 172.21.15.102:0/689841575 conn(0x7ff08c1000e0 msgr2=0x7ff08c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:24.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.317+0000 7ff093000640 1 -- 172.21.15.102:0/689841575 shutdown_connections 2024-04-26T20:50:24.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.317+0000 7ff093000640 1 -- 172.21.15.102:0/689841575 wait complete. 2024-04-26T20:50:24.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.318+0000 7ff093000640 1 Processor -- start 2024-04-26T20:50:24.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.318+0000 7ff093000640 1 -- start start 2024-04-26T20:50:24.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.318+0000 7ff093000640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c104b90 0x7ff08c19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:24.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff093000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff08c105d80 0x7ff08c19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:24.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff093000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff08c106740 0x7ff08c1a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:24.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff093000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c1106e0 con 0x7ff08c106740 2024-04-26T20:50:24.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff093000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c110560 con 0x7ff08c105d80 2024-04-26T20:50:24.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff093000640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c110860 con 0x7ff08c104b90 2024-04-26T20:50:24.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff091000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff08c106740 0x7ff08c1a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:24.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff08be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c104b90 0x7ff08c19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:24.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff08be00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c104b90 0x7ff08c19ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:45442/0 (socket says 172.21.15.102:45442) 2024-04-26T20:50:24.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff08be00640 1 -- 172.21.15.102:0/1543042148 learned_addr learned my addr 172.21.15.102:0/1543042148 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:24.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff08b400640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff08c105d80 0x7ff08c19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:24.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff08be00640 1 -- 172.21.15.102:0/1543042148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff08c105d80 msgr2=0x7ff08c19b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:24.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff08be00640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff08c105d80 0x7ff08c19b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:24.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff08be00640 1 -- 172.21.15.102:0/1543042148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff08c106740 msgr2=0x7ff08c1a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:24.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff08be00640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff08c106740 0x7ff08c1a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:24.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.319+0000 7ff08be00640 1 -- 172.21.15.102:0/1543042148 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08c1a2a30 con 0x7ff08c104b90 2024-04-26T20:50:24.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.320+0000 7ff08be00640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c104b90 0x7ff08c19ae10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff07c031500 tx=0x7ff07c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:24.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.320+0000 7ff088c00640 1 -- 172.21.15.102:0/1543042148 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07c03f070 con 0x7ff08c104b90 2024-04-26T20:50:24.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.320+0000 7ff088c00640 1 -- 172.21.15.102:0/1543042148 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff07c031c50 con 0x7ff08c104b90 2024-04-26T20:50:24.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.320+0000 7ff088c00640 1 -- 172.21.15.102:0/1543042148 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07c035030 con 0x7ff08c104b90 2024-04-26T20:50:24.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.320+0000 7ff093000640 1 -- 172.21.15.102:0/1543042148 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff08c1a2cc0 con 0x7ff08c104b90 2024-04-26T20:50:24.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.320+0000 7ff093000640 1 -- 172.21.15.102:0/1543042148 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff08c1a31b0 con 0x7ff08c104b90 2024-04-26T20:50:24.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.321+0000 7ff093000640 1 -- 172.21.15.102:0/1543042148 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff08c10bf80 con 0x7ff08c104b90 2024-04-26T20:50:24.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.323+0000 7ff088c00640 1 -- 172.21.15.102:0/1543042148 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff07c042070 con 0x7ff08c104b90 2024-04-26T20:50:24.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.324+0000 7ff088c00640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff0700a3c40 0x7ff0700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:24.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.324+0000 7ff088c00640 1 -- 172.21.15.102:0/1543042148 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9609+0+0 (secure 0 0 0) 0x7ff07c0f12b0 con 0x7ff08c104b90 2024-04-26T20:50:24.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.325+0000 7ff08b400640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff0700a3c40 0x7ff0700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:24.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.325+0000 7ff08b400640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff0700a3c40 0x7ff0700a6100 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff08c19c320 tx=0x7ff078005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:24.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.328+0000 7ff088c00640 1 -- 172.21.15.102:0/1543042148 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff07c0bcf20 con 0x7ff08c104b90 2024-04-26T20:50:24.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a[28090]: 2024-04-26T20:50:24.096+0000 7fa2b9600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T20:50:24.399 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d[68484]: ignoring --setuser ceph since I am not root 2024-04-26T20:50:24.399 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d[68484]: ignoring --setgroup ceph since I am not root 2024-04-26T20:50:24.399 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mds[68494]: -- 172.21.15.102:0/377735075 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c4b3381a40 con 0x55c4b3354c00 2024-04-26T20:50:24.399 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d[68484]: starting mds.d at 2024-04-26T20:50:24.400 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g[69117]: ignoring --setuser ceph since I am not root 2024-04-26T20:50:24.400 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g[69117]: ignoring --setgroup ceph since I am not root 2024-04-26T20:50:24.400 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mds[69121]: -- 172.21.15.102:0/1966925086 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555cfc21fa40 con 0x555cfc1f2800 2024-04-26T20:50:24.400 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g[69117]: starting mds.g at 2024-04-26T20:50:24.526 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e[62218]: ignoring --setuser ceph since I am not root 2024-04-26T20:50:24.527 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e[62218]: ignoring --setgroup ceph since I am not root 2024-04-26T20:50:24.527 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e[62218]: starting mds.e at 2024-04-26T20:50:24.527 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h[62534]: ignoring --setuser ceph since I am not root 2024-04-26T20:50:24.527 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h[62534]: ignoring --setgroup ceph since I am not root 2024-04-26T20:50:24.527 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mds[62538]: -- 172.21.15.112:0/2873881635 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c116005a40 con 0x55c115fd9000 2024-04-26T20:50:24.528 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h[62534]: starting mds.h at 2024-04-26T20:50:24.528 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b[62857]: ignoring --setuser ceph since I am not root 2024-04-26T20:50:24.528 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b[62857]: ignoring --setgroup ceph since I am not root 2024-04-26T20:50:24.528 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mds[62861]: -- 172.21.15.112:0/513991745 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5617228c7a40 con 0x56172289a800 2024-04-26T20:50:24.528 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b[62857]: starting mds.b at 2024-04-26T20:50:24.529 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-k[61899]: ignoring --setuser ceph since I am not root 2024-04-26T20:50:24.529 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-k[61899]: ignoring --setgroup ceph since I am not root 2024-04-26T20:50:24.529 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mds[61903]: -- 172.21.15.112:0/1455687637 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cfb88d1a40 con 0x55cfb88a4800 2024-04-26T20:50:24.529 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-k[61899]: starting mds.k at 2024-04-26T20:50:24.565 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f[63037]: ignoring --setuser ceph since I am not root 2024-04-26T20:50:24.566 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f[63037]: ignoring --setgroup ceph since I am not root 2024-04-26T20:50:24.566 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mds[63041]: -- 172.21.15.130:0/2845642310 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5560fdb104e0 con 0x5560fdae6c00 2024-04-26T20:50:24.566 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mds[63041]: -- 172.21.15.130:0/2845642310 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5560fdb13a40 con 0x5560fdae6c00 2024-04-26T20:50:24.566 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f[63037]: starting mds.f at 2024-04-26T20:50:24.566 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c[63351]: ignoring --setuser ceph since I am not root 2024-04-26T20:50:24.567 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c[63351]: ignoring --setgroup ceph since I am not root 2024-04-26T20:50:24.567 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mds[63355]: -- 172.21.15.130:0/1114497229 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5560694441a0 con 0x55606941b000 2024-04-26T20:50:24.567 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c[63351]: starting mds.c at 2024-04-26T20:50:24.567 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i[63662]: ignoring --setuser ceph since I am not root 2024-04-26T20:50:24.567 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i[63662]: ignoring --setgroup ceph since I am not root 2024-04-26T20:50:24.567 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mds[63666]: -- 172.21.15.130:0/1250149121 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f704b9ba40 con 0x55f704b6ec00 2024-04-26T20:50:24.567 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i[63662]: starting mds.i at 2024-04-26T20:50:24.568 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l[62726]: ignoring --setuser ceph since I am not root 2024-04-26T20:50:24.568 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l[62726]: ignoring --setgroup ceph since I am not root 2024-04-26T20:50:24.568 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mds[62730]: -- 172.21.15.130:0/1657958404 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560312e0e4e0 con 0x560312de4800 2024-04-26T20:50:24.568 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mds[62730]: -- 172.21.15.130:0/1657958404 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560312e11a40 con 0x560312de4800 2024-04-26T20:50:24.568 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l[62726]: starting mds.l at 2024-04-26T20:50:24.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:24.577+0000 7ff093000640 1 -- 172.21.15.102:0/1543042148 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff08c02dc80 con 0x7ff08c104b90 2024-04-26T20:50:24.806 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mon[28094]: pgmap v860: 129 pgs: 129 active+clean; 643 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-26T20:50:24.806 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3124766776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:50:24.807 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/2849682965' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:50:24.807 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:50:24.807 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mon[28094]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T20:50:24.807 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mon[28094]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T20:50:24.807 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T20:50:24.807 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mon[28094]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T20:50:24.807 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:24 smithi102 ceph-mon[28094]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-26T20:50:24.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mon[32704]: pgmap v860: 129 pgs: 129 active+clean; 643 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-26T20:50:24.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3124766776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:50:24.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/2849682965' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:50:24.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:50:24.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mon[32704]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T20:50:24.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mon[32704]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T20:50:24.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T20:50:24.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mon[32704]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T20:50:24.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:24 smithi130 ceph-mon[32704]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-26T20:50:25.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mon[32769]: pgmap v860: 129 pgs: 129 active+clean; 643 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-26T20:50:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3124766776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T20:50:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/2849682965' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:50:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T20:50:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mon[32769]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T20:50:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mon[32769]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T20:50:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T20:50:25.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mon[32769]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T20:50:25.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:24 smithi112 ceph-mon[32769]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-26T20:50:25.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.105+0000 7ff088c00640 1 -- 172.21.15.102:0/1543042148 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) ==== 104+0+0 (secure 0 0 0) 0x7ff07c0c1220 con 0x7ff08c104b90 2024-04-26T20:50:25.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.109+0000 7ff093000640 1 -- 172.21.15.102:0/1543042148 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff0700a3c40 msgr2=0x7ff0700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:25.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.109+0000 7ff093000640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff0700a3c40 0x7ff0700a6100 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff08c19c320 tx=0x7ff078005f90 comp rx=0 tx=0).stop 2024-04-26T20:50:25.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.109+0000 7ff093000640 1 -- 172.21.15.102:0/1543042148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c104b90 msgr2=0x7ff08c19ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:25.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.109+0000 7ff093000640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c104b90 0x7ff08c19ae10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff07c031500 tx=0x7ff07c003c10 comp rx=0 tx=0).stop 2024-04-26T20:50:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.110+0000 7ff093000640 1 -- 172.21.15.102:0/1543042148 shutdown_connections 2024-04-26T20:50:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.110+0000 7ff093000640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7ff0700a3c40 0x7ff0700a6100 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.110+0000 7ff093000640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff08c106740 0x7ff08c1a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:25.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.110+0000 7ff093000640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff08c105d80 0x7ff08c19b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:25.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.110+0000 7ff093000640 1 --2- 172.21.15.102:0/1543042148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08c104b90 0x7ff08c19ae10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:25.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.110+0000 7ff093000640 1 -- 172.21.15.102:0/1543042148 >> 172.21.15.102:0/1543042148 conn(0x7ff08c1000e0 msgr2=0x7ff08c109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:25.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.110+0000 7ff093000640 1 -- 172.21.15.102:0/1543042148 shutdown_connections 2024-04-26T20:50:25.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.110+0000 7ff093000640 1 -- 172.21.15.102:0/1543042148 wait complete. 2024-04-26T20:50:25.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T20:50:25.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.316+0000 7fab14a00640 1 -- 172.21.15.102:0/1295654351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab10102900 msgr2=0x7fab10102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:25.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.316+0000 7fab14a00640 1 --2- 172.21.15.102:0/1295654351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab10102900 0x7fab10102d80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faafc009860 tx=0x7faafc02a340 comp rx=0 tx=0).stop 2024-04-26T20:50:25.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.316+0000 7fab14a00640 1 -- 172.21.15.102:0/1295654351 shutdown_connections 2024-04-26T20:50:25.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.316+0000 7fab14a00640 1 --2- 172.21.15.102:0/1295654351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab101032c0 0x7fab10109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:25.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.316+0000 7fab14a00640 1 --2- 172.21.15.102:0/1295654351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab10102900 0x7fab10102d80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:25.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.316+0000 7fab14a00640 1 --2- 172.21.15.102:0/1295654351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab10101710 0x7fab10101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:25.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.316+0000 7fab14a00640 1 -- 172.21.15.102:0/1295654351 >> 172.21.15.102:0/1295654351 conn(0x7fab100fcca0 msgr2=0x7fab100ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:25.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.316+0000 7fab14a00640 1 -- 172.21.15.102:0/1295654351 shutdown_connections 2024-04-26T20:50:25.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.316+0000 7fab14a00640 1 -- 172.21.15.102:0/1295654351 wait complete. 2024-04-26T20:50:25.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.317+0000 7fab14a00640 1 Processor -- start 2024-04-26T20:50:25.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.317+0000 7fab14a00640 1 -- start start 2024-04-26T20:50:25.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.317+0000 7fab14a00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab10101710 0x7fab1010c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:25.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.317+0000 7fab14a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab10102900 0x7fab1010a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:25.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.317+0000 7fab14a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab101032c0 0x7fab1010ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.317+0000 7fab14a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fab1010e360 con 0x7fab10102900 2024-04-26T20:50:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.317+0000 7fab14a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab1010e1e0 con 0x7fab101032c0 2024-04-26T20:50:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.317+0000 7fab14a00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fab1010e4e0 con 0x7fab10101710 2024-04-26T20:50:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.318+0000 7fab0e800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab10101710 0x7fab1010c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.318+0000 7fab0e800640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab10101710 0x7fab1010c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:35740/0 (socket says 172.21.15.102:35740) 2024-04-26T20:50:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.318+0000 7fab0e800640 1 -- 172.21.15.102:0/1208442876 learned_addr learned my addr 172.21.15.102:0/1208442876 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.318+0000 7fab0de00640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab10102900 0x7fab1010a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.318+0000 7fab0f200640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab101032c0 0x7fab1010ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.318+0000 7fab0e800640 1 -- 172.21.15.102:0/1208442876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab101032c0 msgr2=0x7fab1010ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.318+0000 7fab0e800640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab101032c0 0x7fab1010ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.318+0000 7fab0e800640 1 -- 172.21.15.102:0/1208442876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab10102900 msgr2=0x7fab1010a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.318+0000 7fab0e800640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab10102900 0x7fab1010a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.318+0000 7fab0e800640 1 -- 172.21.15.102:0/1208442876 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1010b4a0 con 0x7fab10101710 2024-04-26T20:50:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.319+0000 7fab0e800640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab10101710 0x7fab1010c090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fab0000e9c0 tx=0x7fab0000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.319+0000 7fab07400640 1 -- 172.21.15.102:0/1208442876 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0000cd80 con 0x7fab10101710 2024-04-26T20:50:25.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.319+0000 7fab07400640 1 -- 172.21.15.102:0/1208442876 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab000035f0 con 0x7fab10101710 2024-04-26T20:50:25.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.319+0000 7fab14a00640 1 -- 172.21.15.102:0/1208442876 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab101a2b10 con 0x7fab10101710 2024-04-26T20:50:25.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.319+0000 7fab07400640 1 -- 172.21.15.102:0/1208442876 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00010690 con 0x7fab10101710 2024-04-26T20:50:25.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.319+0000 7fab14a00640 1 -- 172.21.15.102:0/1208442876 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab101a3030 con 0x7fab10101710 2024-04-26T20:50:25.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.320+0000 7fab14a00640 1 -- 172.21.15.102:0/1208442876 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab10069670 con 0x7fab10101710 2024-04-26T20:50:25.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.322+0000 7fab07400640 1 -- 172.21.15.102:0/1208442876 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fab000037a0 con 0x7fab10101710 2024-04-26T20:50:25.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.323+0000 7fab07400640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faaf40a3c40 0x7faaf40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:25.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.324+0000 7fab07400640 1 -- 172.21.15.102:0/1208442876 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9609+0+0 (secure 0 0 0) 0x7fab000d0f80 con 0x7fab10101710 2024-04-26T20:50:25.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.324+0000 7fab0de00640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faaf40a3c40 0x7faaf40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:25.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.324+0000 7fab0de00640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faaf40a3c40 0x7faaf40a6100 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faafc000c00 tx=0x7faafc030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:25.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.326+0000 7fab07400640 1 -- 172.21.15.102:0/1208442876 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fab0009d240 con 0x7fab10101710 2024-04-26T20:50:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:25.530+0000 7fab14a00640 1 -- 172.21.15.102:0/1208442876 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fab10108b00 con 0x7fab10101710 2024-04-26T20:50:25.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1543042148' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:50:25.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:50:25.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T20:50:25.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T20:50:25.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: Cluster is now healthy 2024-04-26T20:50:25.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:50:25.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.112:6832/3238057129,v1:172.21.15.112:6834/3238057129] up:boot 2024-04-26T20:50:25.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.112:6833/2231448108,v1:172.21.15.112:6835/2231448108] up:boot 2024-04-26T20:50:25.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.130:6832/2262125004,v1:172.21.15.130:6836/2262125004] up:boot 2024-04-26T20:50:25.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.130:6835/4220407139,v1:172.21.15.130:6838/4220407139] up:boot 2024-04-26T20:50:25.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.130:6833/3712407648,v1:172.21.15.130:6837/3712407648] up:boot 2024-04-26T20:50:25.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.130:6834/2631831876,v1:172.21.15.130:6839/2631831876] up:boot 2024-04-26T20:50:25.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.112:6836/2547107538,v1:172.21.15.112:6838/2547107538] up:boot 2024-04-26T20:50:25.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.112:6837/2271200085,v1:172.21.15.112:6839/2271200085] up:boot 2024-04-26T20:50:25.873 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.102:6836/535350797,v1:172.21.15.102:6837/535350797] up:boot 2024-04-26T20:50:25.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: mds.? [v2:172.21.15.102:6838/1732786329,v1:172.21.15.102:6839/1732786329] up:boot 2024-04-26T20:50:25.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: fsmap 12 up:standby 2024-04-26T20:50:25.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:50:25.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:50:25.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:50:25.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T20:50:25.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:50:25.874 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T20:50:25.876 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:50:25.877 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T20:50:25.877 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T20:50:25.877 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T20:50:25.877 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/1208442876' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:25.877 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:25 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:25.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1543042148' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:50:25.900 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:50:25.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T20:50:25.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T20:50:25.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: Cluster is now healthy 2024-04-26T20:50:25.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:50:25.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.112:6832/3238057129,v1:172.21.15.112:6834/3238057129] up:boot 2024-04-26T20:50:25.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.112:6833/2231448108,v1:172.21.15.112:6835/2231448108] up:boot 2024-04-26T20:50:25.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.130:6832/2262125004,v1:172.21.15.130:6836/2262125004] up:boot 2024-04-26T20:50:25.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.130:6835/4220407139,v1:172.21.15.130:6838/4220407139] up:boot 2024-04-26T20:50:25.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.130:6833/3712407648,v1:172.21.15.130:6837/3712407648] up:boot 2024-04-26T20:50:25.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.130:6834/2631831876,v1:172.21.15.130:6839/2631831876] up:boot 2024-04-26T20:50:25.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.112:6836/2547107538,v1:172.21.15.112:6838/2547107538] up:boot 2024-04-26T20:50:25.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.112:6837/2271200085,v1:172.21.15.112:6839/2271200085] up:boot 2024-04-26T20:50:25.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.102:6836/535350797,v1:172.21.15.102:6837/535350797] up:boot 2024-04-26T20:50:25.904 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: mds.? [v2:172.21.15.102:6838/1732786329,v1:172.21.15.102:6839/1732786329] up:boot 2024-04-26T20:50:25.904 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: fsmap 12 up:standby 2024-04-26T20:50:25.904 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:50:25.904 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:50:25.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:50:25.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T20:50:25.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:50:25.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T20:50:25.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:50:25.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T20:50:25.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T20:50:25.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T20:50:25.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/1208442876' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:25.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:25 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:26.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1543042148' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:50:26.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T20:50:26.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T20:50:26.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T20:50:26.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: Cluster is now healthy 2024-04-26T20:50:26.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T20:50:26.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.112:6832/3238057129,v1:172.21.15.112:6834/3238057129] up:boot 2024-04-26T20:50:26.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.112:6833/2231448108,v1:172.21.15.112:6835/2231448108] up:boot 2024-04-26T20:50:26.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.130:6832/2262125004,v1:172.21.15.130:6836/2262125004] up:boot 2024-04-26T20:50:26.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.130:6835/4220407139,v1:172.21.15.130:6838/4220407139] up:boot 2024-04-26T20:50:26.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.130:6833/3712407648,v1:172.21.15.130:6837/3712407648] up:boot 2024-04-26T20:50:26.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.130:6834/2631831876,v1:172.21.15.130:6839/2631831876] up:boot 2024-04-26T20:50:26.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.112:6836/2547107538,v1:172.21.15.112:6838/2547107538] up:boot 2024-04-26T20:50:26.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.112:6837/2271200085,v1:172.21.15.112:6839/2271200085] up:boot 2024-04-26T20:50:26.028 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.102:6836/535350797,v1:172.21.15.102:6837/535350797] up:boot 2024-04-26T20:50:26.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: mds.? [v2:172.21.15.102:6838/1732786329,v1:172.21.15.102:6839/1732786329] up:boot 2024-04-26T20:50:26.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: fsmap 12 up:standby 2024-04-26T20:50:26.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T20:50:26.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T20:50:26.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T20:50:26.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T20:50:26.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T20:50:26.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T20:50:26.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T20:50:26.030 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T20:50:26.030 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T20:50:26.030 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='mgr.14152 172.21.15.102:0/2474000047' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T20:50:26.030 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/1208442876' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:26.030 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:25 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:26.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.109+0000 7fab07400640 1 -- 172.21.15.102:0/1208442876 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) ==== 181+0+0 (secure 0 0 0) 0x7fab0009cb40 con 0x7fab10101710 2024-04-26T20:50:26.111 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' removed 2024-04-26T20:50:26.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.116+0000 7fab14a00640 1 -- 172.21.15.102:0/1208442876 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faaf40a3c40 msgr2=0x7faaf40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:26.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.116+0000 7fab14a00640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faaf40a3c40 0x7faaf40a6100 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faafc000c00 tx=0x7faafc030470 comp rx=0 tx=0).stop 2024-04-26T20:50:26.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.116+0000 7fab14a00640 1 -- 172.21.15.102:0/1208442876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab10101710 msgr2=0x7fab1010c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:26.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.116+0000 7fab14a00640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab10101710 0x7fab1010c090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fab0000e9c0 tx=0x7fab0000ee80 comp rx=0 tx=0).stop 2024-04-26T20:50:26.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.117+0000 7fab14a00640 1 -- 172.21.15.102:0/1208442876 shutdown_connections 2024-04-26T20:50:26.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.117+0000 7fab14a00640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7faaf40a3c40 0x7faaf40a6100 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:26.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.117+0000 7fab14a00640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab10102900 0x7fab1010a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:26.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.117+0000 7fab14a00640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab101032c0 0x7fab1010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:26.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.117+0000 7fab14a00640 1 --2- 172.21.15.102:0/1208442876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab10101710 0x7fab1010c090 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:26.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.117+0000 7fab14a00640 1 -- 172.21.15.102:0/1208442876 >> 172.21.15.102:0/1208442876 conn(0x7fab100fcca0 msgr2=0x7fab10105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:26.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.117+0000 7fab14a00640 1 -- 172.21.15.102:0/1208442876 shutdown_connections 2024-04-26T20:50:26.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.117+0000 7fab14a00640 1 -- 172.21.15.102:0/1208442876 wait complete. 2024-04-26T20:50:26.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T20:50:26.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.347+0000 7f1e22c00640 1 -- 172.21.15.102:0/3545396274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e1c105c60 msgr2=0x7f1e1c1060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:26.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.347+0000 7f1e22c00640 1 --2- 172.21.15.102:0/3545396274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e1c105c60 0x7f1e1c1060e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009900 tx=0x7f1e0c02a3f0 comp rx=0 tx=0).stop 2024-04-26T20:50:26.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.347+0000 7f1e22c00640 1 -- 172.21.15.102:0/3545396274 shutdown_connections 2024-04-26T20:50:26.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.347+0000 7f1e22c00640 1 --2- 172.21.15.102:0/3545396274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e1c105c60 0x7f1e1c1060e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:26.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.347+0000 7f1e22c00640 1 --2- 172.21.15.102:0/3545396274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e1c106620 0x7f1e1c10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:26.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.347+0000 7f1e22c00640 1 --2- 172.21.15.102:0/3545396274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e1c104a70 0x7f1e1c104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:26.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.347+0000 7f1e22c00640 1 -- 172.21.15.102:0/3545396274 >> 172.21.15.102:0/3545396274 conn(0x7f1e1c0fffe0 msgr2=0x7f1e1c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:26.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.347+0000 7f1e22c00640 1 -- 172.21.15.102:0/3545396274 shutdown_connections 2024-04-26T20:50:26.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.348+0000 7f1e22c00640 1 -- 172.21.15.102:0/3545396274 wait complete. 2024-04-26T20:50:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.348+0000 7f1e22c00640 1 Processor -- start 2024-04-26T20:50:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e22c00640 1 -- start start 2024-04-26T20:50:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e22c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e1c104a70 0x7f1e1c19ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e22c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e1c105c60 0x7f1e1c19b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e22c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e1c106620 0x7f1e1c1a2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e22c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c1105f0 con 0x7f1e1c105c60 2024-04-26T20:50:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e22c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c110470 con 0x7f1e1c106620 2024-04-26T20:50:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e22c00640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c110770 con 0x7f1e1c104a70 2024-04-26T20:50:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e20e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e1c105c60 0x7f1e1c19b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e22200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e1c106620 0x7f1e1c1a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e22200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e1c106620 0x7f1e1c1a2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.102:37608/0 (socket says 172.21.15.102:37608) 2024-04-26T20:50:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e22200640 1 -- 172.21.15.102:0/642761629 learned_addr learned my addr 172.21.15.102:0/642761629 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:26.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.349+0000 7f1e21800640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e1c104a70 0x7f1e1c19ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.350+0000 7f1e22200640 1 -- 172.21.15.102:0/642761629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e1c104a70 msgr2=0x7f1e1c19ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.350+0000 7f1e22200640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e1c104a70 0x7f1e1c19ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.350+0000 7f1e22200640 1 -- 172.21.15.102:0/642761629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e1c105c60 msgr2=0x7f1e1c19b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.350+0000 7f1e22200640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e1c105c60 0x7f1e1c19b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.350+0000 7f1e22200640 1 -- 172.21.15.102:0/642761629 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1c1a2910 con 0x7f1e1c106620 2024-04-26T20:50:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.350+0000 7f1e22200640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e1c106620 0x7f1e1c1a2270 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400eb10 tx=0x7f1e1400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.350+0000 7f1e1a000640 1 -- 172.21.15.102:0/642761629 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e14004b20 con 0x7f1e1c106620 2024-04-26T20:50:26.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.350+0000 7f1e22c00640 1 -- 172.21.15.102:0/642761629 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e1c1a2c00 con 0x7f1e1c106620 2024-04-26T20:50:26.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.350+0000 7f1e1a000640 1 -- 172.21.15.102:0/642761629 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e14004cc0 con 0x7f1e1c106620 2024-04-26T20:50:26.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.350+0000 7f1e1a000640 1 -- 172.21.15.102:0/642761629 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e14010690 con 0x7f1e1c106620 2024-04-26T20:50:26.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.351+0000 7f1e22c00640 1 -- 172.21.15.102:0/642761629 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e1c1a3150 con 0x7f1e1c106620 2024-04-26T20:50:26.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.351+0000 7f1e22c00640 1 -- 172.21.15.102:0/642761629 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e1c10be60 con 0x7f1e1c106620 2024-04-26T20:50:26.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.354+0000 7f1e1a000640 1 -- 172.21.15.102:0/642761629 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f1e14010830 con 0x7f1e1c106620 2024-04-26T20:50:26.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.355+0000 7f1e1a000640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1e040a3cc0 0x7f1e040a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:26.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.355+0000 7f1e1a000640 1 -- 172.21.15.102:0/642761629 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9055+0+0 (secure 0 0 0) 0x7f1e140d1080 con 0x7f1e1c106620 2024-04-26T20:50:26.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.355+0000 7f1e21800640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1e040a3cc0 0x7f1e040a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:26.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.356+0000 7f1e21800640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1e040a3cc0 0x7f1e040a6180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c19c1d0 tx=0x7f1e10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:26.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.357+0000 7f1e1a000640 1 -- 172.21.15.102:0/642761629 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1e14054060 con 0x7f1e1c106620 2024-04-26T20:50:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:26.553+0000 7f1e22c00640 1 -- 172.21.15.102:0/642761629 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1e1c02dad0 con 0x7f1e1c106620 2024-04-26T20:50:26.871 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:26 smithi130 ceph-mon[32704]: pgmap v862: 129 pgs: 129 active+clean; 644 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 921 B/s wr, 11 op/s 2024-04-26T20:50:26.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:26 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:50:26.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:26 smithi130 ceph-mon[32704]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T20:50:26.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:26 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/642761629' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:26.872 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:26 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:26.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:26 smithi102 ceph-mon[28094]: pgmap v862: 129 pgs: 129 active+clean; 644 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 921 B/s wr, 11 op/s 2024-04-26T20:50:26.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:26 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:50:26.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:26 smithi102 ceph-mon[28094]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T20:50:26.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:26 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/642761629' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:26.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:26 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:27.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:26 smithi112 ceph-mon[32769]: pgmap v862: 129 pgs: 129 active+clean; 644 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 921 B/s wr, 11 op/s 2024-04-26T20:50:27.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:26 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:50:27.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:26 smithi112 ceph-mon[32769]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T20:50:27.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:26 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/642761629' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:27.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:26 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:27.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.112+0000 7f1e1a000640 1 -- 172.21.15.102:0/642761629 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) ==== 169+0+0 (secure 0 0 0) 0x7f1e1409d570 con 0x7f1e1c106620 2024-04-26T20:50:27.114 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' removed 2024-04-26T20:50:27.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.118+0000 7f1e22c00640 1 -- 172.21.15.102:0/642761629 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1e040a3cc0 msgr2=0x7f1e040a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:27.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.118+0000 7f1e22c00640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1e040a3cc0 0x7f1e040a6180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c19c1d0 tx=0x7f1e10005f90 comp rx=0 tx=0).stop 2024-04-26T20:50:27.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.118+0000 7f1e22c00640 1 -- 172.21.15.102:0/642761629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e1c106620 msgr2=0x7f1e1c1a2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:27.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.118+0000 7f1e22c00640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e1c106620 0x7f1e1c1a2270 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400eb10 tx=0x7f1e1400c2d0 comp rx=0 tx=0).stop 2024-04-26T20:50:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.119+0000 7f1e22c00640 1 -- 172.21.15.102:0/642761629 shutdown_connections 2024-04-26T20:50:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.119+0000 7f1e22c00640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f1e040a3cc0 0x7f1e040a6180 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.119+0000 7f1e22c00640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e1c105c60 0x7f1e1c19b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.119+0000 7f1e22c00640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e1c106620 0x7f1e1c1a2270 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.119+0000 7f1e22c00640 1 --2- 172.21.15.102:0/642761629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e1c104a70 0x7f1e1c19ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.119+0000 7f1e22c00640 1 -- 172.21.15.102:0/642761629 >> 172.21.15.102:0/642761629 conn(0x7f1e1c0fffe0 msgr2=0x7f1e1c109030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:27.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.119+0000 7f1e22c00640 1 -- 172.21.15.102:0/642761629 shutdown_connections 2024-04-26T20:50:27.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.119+0000 7f1e22c00640 1 -- 172.21.15.102:0/642761629 wait complete. 2024-04-26T20:50:27.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-26T20:50:27.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.344+0000 7f9ca8600640 1 -- 172.21.15.102:0/515782582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0105d80 msgr2=0x7f9ca0106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.344+0000 7f9ca8600640 1 --2- 172.21.15.102:0/515782582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0105d80 0x7f9ca0106200 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009900 tx=0x7f9c8c02a410 comp rx=0 tx=0).stop 2024-04-26T20:50:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.344+0000 7f9ca8600640 1 -- 172.21.15.102:0/515782582 shutdown_connections 2024-04-26T20:50:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.344+0000 7f9ca8600640 1 --2- 172.21.15.102:0/515782582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0105d80 0x7f9ca0106200 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.344+0000 7f9ca8600640 1 --2- 172.21.15.102:0/515782582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ca0106740 0x7f9ca010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.344+0000 7f9ca8600640 1 --2- 172.21.15.102:0/515782582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ca0104b90 0x7f9ca0104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.344+0000 7f9ca8600640 1 -- 172.21.15.102:0/515782582 >> 172.21.15.102:0/515782582 conn(0x7f9ca0100120 msgr2=0x7f9ca0102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.344+0000 7f9ca8600640 1 -- 172.21.15.102:0/515782582 shutdown_connections 2024-04-26T20:50:27.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.344+0000 7f9ca8600640 1 -- 172.21.15.102:0/515782582 wait complete. 2024-04-26T20:50:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.345+0000 7f9ca8600640 1 Processor -- start 2024-04-26T20:50:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.345+0000 7f9ca8600640 1 -- start start 2024-04-26T20:50:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.345+0000 7f9ca8600640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ca0104b90 0x7f9ca019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.345+0000 7f9ca8600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0105d80 0x7f9ca019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca8600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ca0106740 0x7f9ca01a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca8600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca01106c0 con 0x7f9ca0105d80 2024-04-26T20:50:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca8600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0110540 con 0x7f9ca0106740 2024-04-26T20:50:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca8600640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0110840 con 0x7f9ca0104b90 2024-04-26T20:50:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca5c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ca0104b90 0x7f9ca019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca5200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0105d80 0x7f9ca019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca5c00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ca0104b90 0x7f9ca019adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.102:35778/0 (socket says 172.21.15.102:35778) 2024-04-26T20:50:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca5c00640 1 -- 172.21.15.102:0/3832589164 learned_addr learned my addr 172.21.15.102:0/3832589164 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-26T20:50:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca6600640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ca0106740 0x7f9ca01a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca5c00640 1 -- 172.21.15.102:0/3832589164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ca0106740 msgr2=0x7f9ca01a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca5c00640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ca0106740 0x7f9ca01a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca5c00640 1 -- 172.21.15.102:0/3832589164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0105d80 msgr2=0x7f9ca019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca5c00640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0105d80 0x7f9ca019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca5c00640 1 -- 172.21.15.102:0/3832589164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca01a2990 con 0x7f9ca0104b90 2024-04-26T20:50:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.346+0000 7f9ca5200640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0105d80 0x7f9ca019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T20:50:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.347+0000 7f9ca5c00640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ca0104b90 0x7f9ca019adc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400e9c0 tx=0x7f9c9400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.347+0000 7f9c9e000640 1 -- 172.21.15.102:0/3832589164 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9400cde0 con 0x7f9ca0104b90 2024-04-26T20:50:27.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.347+0000 7f9ca8600640 1 -- 172.21.15.102:0/3832589164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca01a2c80 con 0x7f9ca0104b90 2024-04-26T20:50:27.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.347+0000 7f9c9e000640 1 -- 172.21.15.102:0/3832589164 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c940108e0 con 0x7f9ca0104b90 2024-04-26T20:50:27.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.347+0000 7f9c9e000640 1 -- 172.21.15.102:0/3832589164 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c940188d0 con 0x7f9ca0104b90 2024-04-26T20:50:27.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.347+0000 7f9ca8600640 1 -- 172.21.15.102:0/3832589164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca01a31a0 con 0x7f9ca0104b90 2024-04-26T20:50:27.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.348+0000 7f9ca8600640 1 -- 172.21.15.102:0/3832589164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca010bf80 con 0x7f9ca0104b90 2024-04-26T20:50:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.350+0000 7f9c9e000640 1 -- 172.21.15.102:0/3832589164 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f9c9401f020 con 0x7f9ca0104b90 2024-04-26T20:50:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.351+0000 7f9c9e000640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9c880a3c40 0x7f9c880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T20:50:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.352+0000 7f9c9e000640 1 -- 172.21.15.102:0/3832589164 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8496+0+0 (secure 0 0 0) 0x7f9c940d0e40 con 0x7f9ca0104b90 2024-04-26T20:50:27.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.352+0000 7f9ca5200640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9c880a3c40 0x7f9c880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T20:50:27.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.352+0000 7f9ca5200640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9c880a3c40 0x7f9c880a6100 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c000c00 tx=0x7f9c8c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T20:50:27.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.354+0000 7f9c9e000640 1 -- 172.21.15.102:0/3832589164 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9c9409d580 con 0x7f9ca0104b90 2024-04-26T20:50:27.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:27.543+0000 7f9ca8600640 1 -- 172.21.15.102:0/3832589164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9ca00635d0 con 0x7f9ca0104b90 2024-04-26T20:50:28.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.124+0000 7f9c9e000640 1 -- 172.21.15.102:0/3832589164 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v97) ==== 178+0+0 (secure 0 0 0) 0x7f9c9409ce80 con 0x7f9ca0104b90 2024-04-26T20:50:28.125 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data_ec' removed 2024-04-26T20:50:28.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 -- 172.21.15.102:0/3832589164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9c880a3c40 msgr2=0x7f9c880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:28.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9c880a3c40 0x7f9c880a6100 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c000c00 tx=0x7f9c8c0314a0 comp rx=0 tx=0).stop 2024-04-26T20:50:28.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 -- 172.21.15.102:0/3832589164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ca0104b90 msgr2=0x7f9ca019adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T20:50:28.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ca0104b90 0x7f9ca019adc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400e9c0 tx=0x7f9c9400ee80 comp rx=0 tx=0).stop 2024-04-26T20:50:28.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 -- 172.21.15.102:0/3832589164 shutdown_connections 2024-04-26T20:50:28.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.102:6800/71145250,v1:172.21.15.102:6801/71145250] conn(0x7f9c880a3c40 0x7f9c880a6100 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:28.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca0105d80 0x7f9ca019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:28.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ca0106740 0x7f9ca01a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:28.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 --2- 172.21.15.102:0/3832589164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ca0104b90 0x7f9ca019adc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T20:50:28.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 -- 172.21.15.102:0/3832589164 >> 172.21.15.102:0/3832589164 conn(0x7f9ca0100120 msgr2=0x7f9ca0109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T20:50:28.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.131+0000 7f9ca8600640 1 -- 172.21.15.102:0/3832589164 shutdown_connections 2024-04-26T20:50:28.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-26T20:50:28.132+0000 7f9ca8600640 1 -- 172.21.15.102:0/3832589164 wait complete. 2024-04-26T20:50:28.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:50:28.228 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:50:28.261 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:50:28.294 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T20:50:28.295 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T20:50:28.353 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T20:50:28.371 DEBUG:teuthology.orchestra.run.smithi130:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T20:50:28.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:28 smithi130 ceph-mon[32704]: pgmap v864: 97 pgs: 97 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:50:28.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:28 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:50:28.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:28 smithi130 ceph-mon[32704]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T20:50:28.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:28 smithi130 ceph-mon[32704]: from='client.? 172.21.15.102:0/3832589164' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:28.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:28 smithi130 ceph-mon[32704]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:28.388 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T20:50:28.389 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T20:50:28.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.a 2024-04-26T20:50:28.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:28 smithi102 ceph-mon[28094]: pgmap v864: 97 pgs: 97 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:50:28.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:28 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:50:28.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:28 smithi102 ceph-mon[28094]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T20:50:28.398 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:28 smithi102 ceph-mon[28094]: from='client.? 172.21.15.102:0/3832589164' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:28.399 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:28 smithi102 ceph-mon[28094]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:28.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:28 smithi112 ceph-mon[32769]: pgmap v864: 97 pgs: 97 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T20:50:28.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:28 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T20:50:28.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:28 smithi112 ceph-mon[32769]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T20:50:28.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:28 smithi112 ceph-mon[32769]: from='client.? 172.21.15.102:0/3832589164' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:28.527 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:28 smithi112 ceph-mon[32769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T20:50:28.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:28 smithi102 systemd[1]: Stopping Ceph mon.a for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:50:29.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a[28090]: 2024-04-26T20:50:28.963+0000 7fa2c0400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:50:29.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a[28090]: 2024-04-26T20:50:28.963+0000 7fa2c0400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-26T20:50:29.306 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:28 smithi102 podman[85566]: 2024-04-26 20:50:28.994665206 +0000 UTC m=+0.162582435 container died 747c74129f19a9156b2e006f7432e9b4ab52326e07e7bebb0b5eb6ec64806570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-26T20:50:29.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:29 smithi102 podman[85566]: 2024-04-26 20:50:29.590675987 +0000 UTC m=+0.758593224 container cleanup 747c74129f19a9156b2e006f7432e9b4ab52326e07e7bebb0b5eb6ec64806570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0) 2024-04-26T20:50:29.899 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:29 smithi102 bash[85566]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a 2024-04-26T20:50:30.183 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 26 20:50:29 smithi102 podman[85580]: 2024-04-26 20:50:29.909462587 +0000 UTC m=+0.900395511 container remove 747c74129f19a9156b2e006f7432e9b4ab52326e07e7bebb0b5eb6ec64806570 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T20:50:30.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T20:50:30.417 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:50:30.417 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T20:50:30.417 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-26T20:50:30.417 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.b 2024-04-26T20:50:30.750 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:30 smithi112 systemd[1]: Stopping Ceph mon.b for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:50:31.026 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:30 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-b[32764]: 2024-04-26T20:50:30.749+0000 7f0d4a8c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:50:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:30 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-b[32764]: 2024-04-26T20:50:30.749+0000 7f0d4a8c8700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-26T20:50:31.027 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:30 smithi112 podman[67200]: 2024-04-26 20:50:30.873131455 +0000 UTC m=+0.269141477 container died 8b3a8b0d25b783d8697fdfaaf2f46fd41e2ec8a0ff5043b15c470834661f82e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T20:50:31.727 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:31 smithi112 podman[67200]: 2024-04-26 20:50:31.517777586 +0000 UTC m=+0.913787602 container cleanup 8b3a8b0d25b783d8697fdfaaf2f46fd41e2ec8a0ff5043b15c470834661f82e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:50:31.728 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:31 smithi112 bash[67200]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-b 2024-04-26T20:50:31.977 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:31 smithi112 podman[67214]: 2024-04-26 20:50:31.727718345 +0000 UTC m=+0.849058183 container remove 8b3a8b0d25b783d8697fdfaaf2f46fd41e2ec8a0ff5043b15c470834661f82e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:50:31.978 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:31 smithi112 systemd[1]: ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-26T20:50:31.978 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:31 smithi112 systemd[1]: Stopped Ceph mon.b for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:50:31.978 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 26 20:50:31 smithi112 systemd[1]: ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.b.service: Consumed 14.725s CPU time. 2024-04-26T20:50:31.980 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.b.service' 2024-04-26T20:50:32.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:50:32.051 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-26T20:50:32.051 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-26T20:50:32.052 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.c 2024-04-26T20:50:32.356 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:32 smithi130 systemd[1]: Stopping Ceph mon.c for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:50:32.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:32 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-c[32698]: 2024-04-26T20:50:32.355+0000 7fb3a3452700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:50:32.621 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:32 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-c[32698]: 2024-04-26T20:50:32.355+0000 7fb3a3452700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-26T20:50:32.622 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:32 smithi130 podman[68023]: 2024-04-26 20:50:32.458427349 +0000 UTC m=+0.209479280 container died a8d92a4102eaaec9d97e52aabe65f8b18647b21a56a20ab10ecf398d8dd41dc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T20:50:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:32 smithi130 podman[68023]: 2024-04-26 20:50:32.973837143 +0000 UTC m=+0.724889118 container cleanup a8d92a4102eaaec9d97e52aabe65f8b18647b21a56a20ab10ecf398d8dd41dc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T20:50:33.121 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 26 20:50:32 smithi130 bash[68023]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mon-c 2024-04-26T20:50:33.344 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mon.c.service' 2024-04-26T20:50:33.391 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:50:33.392 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-26T20:50:33.392 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-26T20:50:33.392 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.x 2024-04-26T20:50:33.898 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:50:33 smithi102 systemd[1]: Stopping Ceph mgr.x for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:50:34.354 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:50:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:50:33.964+0000 7fc358e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:50:34.355 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:50:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x[28326]: 2024-04-26T20:50:33.964+0000 7fc358e00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T20:50:34.355 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:50:34 smithi102 podman[85689]: 2024-04-26 20:50:34.058109967 +0000 UTC m=+0.260621047 container died 20c510044d08a23fb5c0733dee46abd56944768c0aa11840b6651674f8246687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-26T20:50:34.649 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:50:34 smithi102 podman[85689]: 2024-04-26 20:50:34.538246312 +0000 UTC m=+0.740757388 container cleanup 20c510044d08a23fb5c0733dee46abd56944768c0aa11840b6651674f8246687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T20:50:34.649 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:50:34 smithi102 bash[85689]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x 2024-04-26T20:50:35.140 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 26 20:50:34 smithi102 podman[85703]: 2024-04-26 20:50:34.723166927 +0000 UTC m=+0.649165090 container remove 20c510044d08a23fb5c0733dee46abd56944768c0aa11840b6651674f8246687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T20:50:35.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-26T20:50:35.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:50:35.268 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-26T20:50:35.269 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-26T20:50:35.269 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.y 2024-04-26T20:50:35.610 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:50:35 smithi112 systemd[1]: Stopping Ceph mgr.y for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:50:35.915 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:50:35 smithi112 podman[67324]: 2024-04-26 20:50:35.610618683 +0000 UTC m=+0.154563469 container died 95b0502d8a59a26e0844e908eb8c64f9b14dbcb55638761bd2d6efe9403cb345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:50:36.276 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:50:36 smithi112 podman[67324]: 2024-04-26 20:50:36.015117388 +0000 UTC m=+0.559062235 container cleanup 95b0502d8a59a26e0844e908eb8c64f9b14dbcb55638761bd2d6efe9403cb345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:50:36.276 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 26 20:50:36 smithi112 bash[67324]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-y 2024-04-26T20:50:36.479 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-26T20:50:36.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:50:36.512 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-26T20:50:36.512 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-26T20:50:36.513 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.z 2024-04-26T20:50:36.858 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:50:36 smithi130 systemd[1]: Stopping Ceph mgr.z for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:50:37.121 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:50:36 smithi130 podman[68146]: 2024-04-26 20:50:36.858216845 +0000 UTC m=+0.147042503 container died 92d42ddf2dc52f60fbba0ceba9ddde80f2c0e7070e5fdafbd583a53d81a9c884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:50:37.531 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:50:37 smithi130 podman[68146]: 2024-04-26 20:50:37.246360222 +0000 UTC m=+0.535185880 container cleanup 92d42ddf2dc52f60fbba0ceba9ddde80f2c0e7070e5fdafbd583a53d81a9c884 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T20:50:37.531 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 26 20:50:37 smithi130 bash[68146]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mgr-z 2024-04-26T20:50:37.667 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-26T20:50:37.699 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:50:37.699 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-26T20:50:37.699 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T20:50:37.699 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.0 2024-04-26T20:50:38.148 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:37 smithi102 systemd[1]: Stopping Ceph osd.0 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:50:38.649 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:38 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0[42540]: 2024-04-26T20:50:38.249+0000 7f9878000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:50:38.649 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:38 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0[42540]: 2024-04-26T20:50:38.249+0000 7f9878000700 -1 osd.0 97 *** Got signal Terminated *** 2024-04-26T20:50:38.649 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:38 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0[42540]: 2024-04-26T20:50:38.249+0000 7f9878000700 -1 osd.0 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:50:44.028 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:43 smithi102 podman[85812]: 2024-04-26 20:50:43.678453563 +0000 UTC m=+5.562464788 container died 3a32bdd8e34a4973a39814f97049aad4e7dd1d251a45aa34bc93a658b835c360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-26T20:50:44.379 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:44 smithi102 podman[85812]: 2024-04-26 20:50:44.151877196 +0000 UTC m=+6.035888460 container cleanup 3a32bdd8e34a4973a39814f97049aad4e7dd1d251a45aa34bc93a658b835c360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-26T20:50:44.379 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:44 smithi102 bash[85812]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0 2024-04-26T20:50:44.648 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:44 smithi102 podman[85831]: 2024-04-26 20:50:44.378221988 +0000 UTC m=+0.680348303 container remove 3a32bdd8e34a4973a39814f97049aad4e7dd1d251a45aa34bc93a658b835c360 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-26T20:50:45.543 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:45 smithi102 podman[85918]: 2024-04-26 20:50:45.129262175 +0000 UTC m=+0.038134525 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:50:45.543 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:45 smithi102 podman[85918]: 2024-04-26 20:50:45.262687618 +0000 UTC m=+0.171559972 container create 7e4e3ff82b8a82e31745bd5950025181ed5572a5eccb73e5b3a5bd19975bd55e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:50:45.543 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:45 smithi102 podman[85918]: 2024-04-26 20:50:45.541605217 +0000 UTC m=+0.450477573 container init 7e4e3ff82b8a82e31745bd5950025181ed5572a5eccb73e5b3a5bd19975bd55e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-04-26T20:50:45.898 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:45 smithi102 podman[85918]: 2024-04-26 20:50:45.554072285 +0000 UTC m=+0.462944610 container start 7e4e3ff82b8a82e31745bd5950025181ed5572a5eccb73e5b3a5bd19975bd55e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:50:45.899 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:45 smithi102 podman[85918]: 2024-04-26 20:50:45.682511768 +0000 UTC m=+0.591384165 container attach 7e4e3ff82b8a82e31745bd5950025181ed5572a5eccb73e5b3a5bd19975bd55e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-04-26T20:50:46.377 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 26 20:50:46 smithi102 podman[85918]: 2024-04-26 20:50:46.002550343 +0000 UTC m=+0.911422713 container died 7e4e3ff82b8a82e31745bd5950025181ed5572a5eccb73e5b3a5bd19975bd55e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:50:46.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T20:50:46.962 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:50:46.962 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T20:50:46.962 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T20:50:46.963 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.1 2024-04-26T20:50:47.276 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:47 smithi112 systemd[1]: Stopping Ceph osd.1 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:50:47.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:47 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1[37369]: 2024-04-26T20:50:47.278+0000 7f1ff8a7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:50:47.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:47 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1[37369]: 2024-04-26T20:50:47.278+0000 7f1ff8a7f700 -1 osd.1 97 *** Got signal Terminated *** 2024-04-26T20:50:47.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:47 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1[37369]: 2024-04-26T20:50:47.278+0000 7f1ff8a7f700 -1 osd.1 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:50:53.012 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:52 smithi112 podman[67448]: 2024-04-26 20:50:52.736073475 +0000 UTC m=+5.580406534 container died b79e65a3ab0829512dde21c63f6ced0543ff55713764613d31ea3f636cdd2183 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-26T20:50:53.276 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:53 smithi112 podman[67448]: 2024-04-26 20:50:53.10347344 +0000 UTC m=+5.947806497 container cleanup b79e65a3ab0829512dde21c63f6ced0543ff55713764613d31ea3f636cdd2183 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:50:53.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:53 smithi112 bash[67448]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1 2024-04-26T20:50:53.716 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:53 smithi112 podman[67466]: 2024-04-26 20:50:53.388293723 +0000 UTC m=+0.646193692 container remove b79e65a3ab0829512dde21c63f6ced0543ff55713764613d31ea3f636cdd2183 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-26T20:50:53.975 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:53 smithi112 podman[67563]: 2024-04-26 20:50:53.619749951 +0000 UTC m=+0.017743815 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:50:53.976 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:53 smithi112 podman[67563]: 2024-04-26 20:50:53.747922779 +0000 UTC m=+0.145916669 container create 3af6d12f00a94e2865a23f4b20e3242d1948bcae8e46fd8d1f3ed2e686acde5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-04-26T20:50:54.269 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:53 smithi112 podman[67563]: 2024-04-26 20:50:53.97546652 +0000 UTC m=+0.373460367 container init 3af6d12f00a94e2865a23f4b20e3242d1948bcae8e46fd8d1f3ed2e686acde5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-26T20:50:54.269 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:53 smithi112 podman[67563]: 2024-04-26 20:50:53.979496187 +0000 UTC m=+0.377490037 container start 3af6d12f00a94e2865a23f4b20e3242d1948bcae8e46fd8d1f3ed2e686acde5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-04-26T20:50:54.269 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:54 smithi112 podman[67563]: 2024-04-26 20:50:54.075640142 +0000 UTC m=+0.473633991 container attach 3af6d12f00a94e2865a23f4b20e3242d1948bcae8e46fd8d1f3ed2e686acde5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T20:50:54.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 26 20:50:54 smithi112 podman[67563]: 2024-04-26 20:50:54.269383118 +0000 UTC m=+0.667376971 container died 3af6d12f00a94e2865a23f4b20e3242d1948bcae8e46fd8d1f3ed2e686acde5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-26T20:50:54.962 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T20:50:54.998 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:50:55.001 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T20:50:55.001 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T20:50:55.002 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.2 2024-04-26T20:50:55.371 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:50:55 smithi130 systemd[1]: Stopping Ceph osd.2 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:50:55.871 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:50:55 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2[37838]: 2024-04-26T20:50:55.491+0000 7f0512f62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:50:55.871 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:50:55 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2[37838]: 2024-04-26T20:50:55.491+0000 7f0512f62700 -1 osd.2 97 *** Got signal Terminated *** 2024-04-26T20:50:55.872 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:50:55 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2[37838]: 2024-04-26T20:50:55.492+0000 7f0512f62700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:51:01.111 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:51:00 smithi130 podman[68287]: 2024-04-26 20:51:00.811108048 +0000 UTC m=+5.626304398 container died faa050cbbe8fb289ec57804ccac376428ffb6215a7094c3ef94cf5532c4ec82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:51:01.372 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:51:01 smithi130 podman[68287]: 2024-04-26 20:51:01.21160298 +0000 UTC m=+6.026799327 container cleanup faa050cbbe8fb289ec57804ccac376428ffb6215a7094c3ef94cf5532c4ec82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T20:51:01.372 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:51:01 smithi130 bash[68287]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2 2024-04-26T20:51:01.776 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:51:01 smithi130 podman[68305]: 2024-04-26 20:51:01.478010988 +0000 UTC m=+0.661035334 container remove faa050cbbe8fb289ec57804ccac376428ffb6215a7094c3ef94cf5532c4ec82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T20:51:02.065 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:51:01 smithi130 podman[68395]: 2024-04-26 20:51:01.679609423 +0000 UTC m=+0.018003893 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:51:02.065 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:51:01 smithi130 podman[68395]: 2024-04-26 20:51:01.787176742 +0000 UTC m=+0.125571228 container create 22ed534e9669ac0bd1a2233ec02cfa2c6fe7a33156de37e59d75c4466201e7cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-04-26T20:51:02.065 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:51:01 smithi130 podman[68395]: 2024-04-26 20:51:01.964698576 +0000 UTC m=+0.303093032 container init 22ed534e9669ac0bd1a2233ec02cfa2c6fe7a33156de37e59d75c4466201e7cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:51:02.066 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:51:01 smithi130 podman[68395]: 2024-04-26 20:51:01.967723976 +0000 UTC m=+0.306118419 container start 22ed534e9669ac0bd1a2233ec02cfa2c6fe7a33156de37e59d75c4466201e7cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T20:51:02.066 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:51:02 smithi130 podman[68395]: 2024-04-26 20:51:02.064929496 +0000 UTC m=+0.403323942 container attach 22ed534e9669ac0bd1a2233ec02cfa2c6fe7a33156de37e59d75c4466201e7cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:51:02.371 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 26 20:51:02 smithi130 podman[68395]: 2024-04-26 20:51:02.2618148 +0000 UTC m=+0.600209253 container died 22ed534e9669ac0bd1a2233ec02cfa2c6fe7a33156de37e59d75c4466201e7cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T20:51:02.925 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T20:51:02.961 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:51:02.961 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T20:51:02.962 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-26T20:51:02.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.3 2024-04-26T20:51:03.368 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:03 smithi102 systemd[1]: Stopping Ceph osd.3 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:51:03.649 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:03 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3[49465]: 2024-04-26T20:51:03.541+0000 7f8bf0800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:51:03.649 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:03 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3[49465]: 2024-04-26T20:51:03.541+0000 7f8bf0800700 -1 osd.3 97 *** Got signal Terminated *** 2024-04-26T20:51:03.649 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:03 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3[49465]: 2024-04-26T20:51:03.541+0000 7f8bf0800700 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:51:04.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:04 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:04.534+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:05.068 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:04 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:04.880+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:05.276 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:04 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:04.843+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:05.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:05.053+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:05.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:05.506+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:06.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:05 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:05.843+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:06.276 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:05 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:05.830+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:06.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:06 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:06.011+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:06.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:06 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:06.701+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:06.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:06 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:06.459+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:07.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:06 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:06.872+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:07.276 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:06 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:06.782+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:07.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:07 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:07.059+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:07.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:07 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:07.663+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:07.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:07 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:07.507+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:08.026 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:07 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:07.764+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:08.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:07 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:07.828+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:08.149 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:07 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3[49465]: 2024-04-26T20:51:07.878+0000 7f8bec000700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:42.917867+0000 front 2024-04-26T20:50:42.918096+0000 (oldest deadline 2024-04-26T20:51:07.018227+0000) 2024-04-26T20:51:08.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:08.053+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:08.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:08.476+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:08.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:08.672+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:09.077 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:08 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:08.790+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:09.276 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:08 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:08.814+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:09.362 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:09 smithi102 podman[86022]: 2024-04-26 20:51:09.076955001 +0000 UTC m=+5.711969893 container died 4a80dc5ed5bc86fd3d42cda78ead1a3465af1043ef1170add73ca3d5c184d6d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T20:51:09.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:09.035+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:09.649 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:09 smithi102 podman[86022]: 2024-04-26 20:51:09.489689369 +0000 UTC m=+6.124704278 container cleanup 4a80dc5ed5bc86fd3d42cda78ead1a3465af1043ef1170add73ca3d5c184d6d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T20:51:09.649 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:09 smithi102 bash[86022]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3 2024-04-26T20:51:09.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:09.638+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:09.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:09.471+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:10.149 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:09 smithi102 podman[86053]: 2024-04-26 20:51:09.699296275 +0000 UTC m=+0.602843828 container remove 4a80dc5ed5bc86fd3d42cda78ead1a3465af1043ef1170add73ca3d5c184d6d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T20:51:10.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:09 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:09.752+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:10.276 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:09 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:09.804+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:10.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:10.011+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:10.852 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:10 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:10.761+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:10.853 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:10 smithi102 podman[86141]: 2024-04-26 20:51:10.500611341 +0000 UTC m=+0.038147015 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:51:10.853 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:10 smithi102 podman[86141]: 2024-04-26 20:51:10.625303375 +0000 UTC m=+0.162839031 container create ab8db53e357dabdb0822bf41ce265d283e28c90f7302472dc88804f907fb90e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:51:10.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:10.478+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:10.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:10.648+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:11.276 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:10 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:10.785+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:11.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:10.996+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:11.399 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:10 smithi102 podman[86141]: 2024-04-26 20:51:10.962685565 +0000 UTC m=+0.500221261 container init ab8db53e357dabdb0822bf41ce265d283e28c90f7302472dc88804f907fb90e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T20:51:11.399 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:10 smithi102 podman[86141]: 2024-04-26 20:51:10.976318715 +0000 UTC m=+0.513854378 container start ab8db53e357dabdb0822bf41ce265d283e28c90f7302472dc88804f907fb90e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T20:51:11.399 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:11 smithi102 podman[86141]: 2024-04-26 20:51:11.053552291 +0000 UTC m=+0.591087943 container attach ab8db53e357dabdb0822bf41ce265d283e28c90f7302472dc88804f907fb90e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-26T20:51:11.794 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 26 20:51:11 smithi102 podman[86141]: 2024-04-26 20:51:11.425733699 +0000 UTC m=+0.963269362 container died ab8db53e357dabdb0822bf41ce265d283e28c90f7302472dc88804f907fb90e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T20:51:11.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:11 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:11.612+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:11.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:11 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:11.516+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:12.116 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:11 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:11.797+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:12.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.3.service' 2024-04-26T20:51:12.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:51:12.270 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-26T20:51:12.270 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-26T20:51:12.270 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.4 2024-04-26T20:51:12.276 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:11 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:11.832+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:12.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:11 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:11.987+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:12.558 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:12 smithi112 systemd[1]: Stopping Ceph osd.4 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:51:12.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:12 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:12.593+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:12.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:12 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:12.515+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:12.877 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:12 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4[43865]: 2024-04-26T20:51:12.557+0000 7f18aff61700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:51:12.877 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:12 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4[43865]: 2024-04-26T20:51:12.557+0000 7f18aff61700 -1 osd.4 97 *** Got signal Terminated *** 2024-04-26T20:51:12.877 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:12 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4[43865]: 2024-04-26T20:51:12.557+0000 7f18aff61700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:51:13.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:12 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:12.758+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:13.276 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:13 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4[43865]: 2024-04-26T20:51:13.100+0000 7f18a854d700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:49.479972+0000 front 2024-04-26T20:50:49.479997+0000 (oldest deadline 2024-04-26T20:51:12.980128+0000) 2024-04-26T20:51:13.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:12 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:12.877+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:13.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:13.018+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:13.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:13.552+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:13.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:13.521+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:14.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:13 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:13.747+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:14.276 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:14 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4[43865]: 2024-04-26T20:51:14.121+0000 7f18a854d700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:49.479972+0000 front 2024-04-26T20:50:49.479997+0000 (oldest deadline 2024-04-26T20:51:12.980128+0000) 2024-04-26T20:51:14.276 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:13 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:13.896+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:14.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:13 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:13.896+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:14.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:14.038+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:14.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:14.524+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:14.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:14.514+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:15.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:14 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:14.673+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:15.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:14 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:14.860+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:15.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:14 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:14.860+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:15.102 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:14 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:14.727+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:15.103 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:14 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:14.727+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:15.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:14.998+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:15.398 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:15 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:15.101+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:15.526 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:15 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4[43865]: 2024-04-26T20:51:15.137+0000 7f18a854d700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:49.479972+0000 front 2024-04-26T20:50:49.479997+0000 (oldest deadline 2024-04-26T20:51:12.980128+0000) 2024-04-26T20:51:15.646 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:15.469+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:15.647 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:15.571+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:16.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:15 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:15.715+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:16.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:15 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:15.827+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:16.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:15 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:15.827+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:16.099 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:15 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:15.745+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:16.100 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:15 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:15.745+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:16.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:15.966+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:16.398 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:16 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:16.098+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:16.526 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:16 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4[43865]: 2024-04-26T20:51:16.134+0000 7f18a854d700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:49.479972+0000 front 2024-04-26T20:50:49.479997+0000 (oldest deadline 2024-04-26T20:51:12.980128+0000) 2024-04-26T20:51:16.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:16 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:16.435+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:16.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:16 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:16.533+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:17.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:16 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:16.761+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:17.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:16 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:16.824+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:17.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:16 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:16.824+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:17.107 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:16 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:16.711+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:17.107 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:16 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:16.711+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:17.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:16 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:16.925+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:17.398 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:17 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:17.105+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:17.526 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:17 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4[43865]: 2024-04-26T20:51:17.156+0000 7f18a854d700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:49.479972+0000 front 2024-04-26T20:50:49.479997+0000 (oldest deadline 2024-04-26T20:51:12.980128+0000) 2024-04-26T20:51:17.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:17 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:17.491+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:17.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:17 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:17.481+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:17.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:17 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:17.481+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:18.027 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:17 smithi112 podman[67681]: 2024-04-26 20:51:17.868245424 +0000 UTC m=+5.402758917 container died 506a2ce608b946c9b68653991ff2eb64b5d22751670d68f45a70199624d05f6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T20:51:18.027 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:17 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:17.749+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:18.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:17 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:17.871+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:18.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:17 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:17.871+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:18.076 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:17 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:17.740+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:18.076 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:17 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:17.740+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:18.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:17 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:17.952+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:18.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:17 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:17.952+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:52.026654+0000 front 2024-04-26T20:50:52.026504+0000 (oldest deadline 2024-04-26T20:51:17.326486+0000) 2024-04-26T20:51:18.398 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:18 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:18.074+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:18.626 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:18 smithi112 podman[67681]: 2024-04-26 20:51:18.391449439 +0000 UTC m=+5.925962927 container cleanup 506a2ce608b946c9b68653991ff2eb64b5d22751670d68f45a70199624d05f6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T20:51:18.626 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:18 smithi112 bash[67681]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4 2024-04-26T20:51:18.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:18 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:18.464+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:18.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:18 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:18.481+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:18.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:18 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:18.481+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:18.909 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:18 smithi112 podman[67699]: 2024-04-26 20:51:18.626115726 +0000 UTC m=+0.751896664 container remove 506a2ce608b946c9b68653991ff2eb64b5d22751670d68f45a70199624d05f6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:51:18.909 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:18 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:18.706+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:19.090 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:18 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:18.736+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:19.091 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:18 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:18.736+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:19.202 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:18 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:18.907+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:19.203 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:18 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:18.907+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:19.203 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:19 smithi112 podman[67787]: 2024-04-26 20:51:18.97019095 +0000 UTC m=+0.017550862 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:51:19.203 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:19 smithi112 podman[67787]: 2024-04-26 20:51:19.093772517 +0000 UTC m=+0.141132451 container create 7ee0ade93aba6dd1f881245cfb3dafb920d4c3683693d29c0008437086b32f25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-26T20:51:19.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:18 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:18.987+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:19.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:18 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:18.987+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:52.026654+0000 front 2024-04-26T20:50:52.026504+0000 (oldest deadline 2024-04-26T20:51:17.326486+0000) 2024-04-26T20:51:19.398 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:19 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:19.089+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:19.527 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:19 smithi112 podman[67787]: 2024-04-26 20:51:19.279729872 +0000 UTC m=+0.327089769 container init 7ee0ade93aba6dd1f881245cfb3dafb920d4c3683693d29c0008437086b32f25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T20:51:19.527 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:19 smithi112 podman[67787]: 2024-04-26 20:51:19.282713205 +0000 UTC m=+0.330073102 container start 7ee0ade93aba6dd1f881245cfb3dafb920d4c3683693d29c0008437086b32f25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T20:51:19.527 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:19 smithi112 podman[67787]: 2024-04-26 20:51:19.371454817 +0000 UTC m=+0.418814708 container attach 7ee0ade93aba6dd1f881245cfb3dafb920d4c3683693d29c0008437086b32f25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T20:51:19.862 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:19 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:19.697+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:19.863 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 26 20:51:19 smithi112 podman[67787]: 2024-04-26 20:51:19.588010618 +0000 UTC m=+0.635370522 container died 7ee0ade93aba6dd1f881245cfb3dafb920d4c3683693d29c0008437086b32f25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-26T20:51:19.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:19 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:19.513+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:19.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:19 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:19.449+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:19.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:19 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:19.449+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:20.109 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:19 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:19.717+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:20.110 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:19 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:19.717+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:20.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:19 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:19.862+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:20.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:19 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:19.862+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:20.233 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.4.service' 2024-04-26T20:51:20.312 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:51:20.312 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-26T20:51:20.313 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-26T20:51:20.313 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.5 2024-04-26T20:51:20.340 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:19 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:19.943+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:20.340 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:19 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:19.943+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:52.026654+0000 front 2024-04-26T20:50:52.026504+0000 (oldest deadline 2024-04-26T20:51:17.326486+0000) 2024-04-26T20:51:20.398 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:20 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:20.108+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:20.605 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:20 smithi130 systemd[1]: Stopping Ceph osd.5 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:51:20.605 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:20 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:20.506+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:20.606 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:20 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:20.470+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:20.606 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:20 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:20.470+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:20.871 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:20 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:20.604+0000 7f59c55c4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:51:20.871 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:20 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:20.604+0000 7f59c55c4700 -1 osd.5 97 *** Got signal Terminated *** 2024-04-26T20:51:20.872 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:20 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:20.604+0000 7f59c55c4700 -1 osd.5 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:51:21.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:20 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:20.705+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:21.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:20 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:20.856+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:21.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:20 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:20.856+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:21.145 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:20 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:20.696+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:21.146 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:20 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:20.696+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:21.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:20 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:20.947+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:21.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:20 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:20.947+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:52.026654+0000 front 2024-04-26T20:50:52.026504+0000 (oldest deadline 2024-04-26T20:51:17.326486+0000) 2024-04-26T20:51:21.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:20 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:20.947+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:57.326852+0000 front 2024-04-26T20:50:57.326802+0000 (oldest deadline 2024-04-26T20:51:20.826998+0000) 2024-04-26T20:51:21.398 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:21 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:21.144+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:21.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:21 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:21.546+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:21.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:21 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:21.469+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:21.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:21 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:21.469+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:21.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:21 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:21.469+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:21.898 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:21 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:21.718+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:21.899 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:21 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:21.718+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:22.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:21 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:21.749+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:22.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:21 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:21.830+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:22.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:21 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:21.830+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:22.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:21 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:21.994+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:22.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:21 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:21.994+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:52.026654+0000 front 2024-04-26T20:50:52.026504+0000 (oldest deadline 2024-04-26T20:51:17.326486+0000) 2024-04-26T20:51:22.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:21 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:21.994+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:57.326852+0000 front 2024-04-26T20:50:57.326802+0000 (oldest deadline 2024-04-26T20:51:20.826998+0000) 2024-04-26T20:51:22.648 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:22 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:22.179+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:22.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:22 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:22.578+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:22.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:22 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:22.517+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:22.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:22 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:22.517+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:22.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:22 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:22.517+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:23.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:22 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:22.721+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:23.027 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:22 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:22.721+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:23.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:22 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:22.877+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:23.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:22 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:22.877+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:23.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:22 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:22.877+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:23.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:22 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:22.762+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:23.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:22 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:22.762+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:23.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:22 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:22.762+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:23.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:22 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:22.946+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:23.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:22 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:22.946+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:52.026654+0000 front 2024-04-26T20:50:52.026504+0000 (oldest deadline 2024-04-26T20:51:17.326486+0000) 2024-04-26T20:51:23.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:22 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:22.946+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:57.326852+0000 front 2024-04-26T20:50:57.326802+0000 (oldest deadline 2024-04-26T20:51:20.826998+0000) 2024-04-26T20:51:23.648 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:23 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:23.159+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:23.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:23 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:23.589+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:23.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:23 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:23.511+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:23.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:23 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:23.511+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:23.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:23 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:23.511+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:24.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:23 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:23.769+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:24.027 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:23 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:23.769+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:24.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:23 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:23.902+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:24.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:23 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:23.902+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:24.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:23 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:23.902+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:24.120 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:23 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:23.787+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:24.121 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:23 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:23.787+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:24.121 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:23 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:23.787+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:24.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:23 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:23.905+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:24.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:23 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:23.905+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:52.026654+0000 front 2024-04-26T20:50:52.026504+0000 (oldest deadline 2024-04-26T20:51:17.326486+0000) 2024-04-26T20:51:24.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:23 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:23.905+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:57.326852+0000 front 2024-04-26T20:50:57.326802+0000 (oldest deadline 2024-04-26T20:51:20.826998+0000) 2024-04-26T20:51:24.398 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:24.119+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:24.398 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:24.119+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.086264+0000 front 2024-04-26T20:50:59.086205+0000 (oldest deadline 2024-04-26T20:51:23.786306+0000) 2024-04-26T20:51:24.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:24.545+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:24.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:24.545+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:24.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:24.541+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:24.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:24.541+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:24.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:24.541+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:25.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:24.763+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:25.027 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:24.763+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:25.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:24.901+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:25.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:24.901+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:25.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:24 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:24.901+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:25.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:24.763+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:25.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:24.763+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:25.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:24 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:24.763+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:25.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:24.923+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:40.925513+0000 front 2024-04-26T20:50:40.925200+0000 (oldest deadline 2024-04-26T20:51:04.424891+0000) 2024-04-26T20:51:25.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:24.923+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:52.026654+0000 front 2024-04-26T20:50:52.026504+0000 (oldest deadline 2024-04-26T20:51:17.326486+0000) 2024-04-26T20:51:25.372 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:24 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5[44788]: 2024-04-26T20:51:24.923+0000 7f59c1bb8700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:57.326852+0000 front 2024-04-26T20:50:57.326802+0000 (oldest deadline 2024-04-26T20:51:20.826998+0000) 2024-04-26T20:51:25.648 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:25 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:25.168+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:25.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:25 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:25.168+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.086264+0000 front 2024-04-26T20:50:59.086205+0000 (oldest deadline 2024-04-26T20:51:23.786306+0000) 2024-04-26T20:51:25.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:25 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:25.570+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:25.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:25 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:25.570+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:25.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:25 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:25.590+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:25.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:25 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:25.590+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:25.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:25 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:25.590+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:26.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:25 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:25.715+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:26.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:25 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:25.715+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:26.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:25 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:25.715+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:26.195 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:25 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:25.806+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:26.196 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:25 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:25.806+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:26.196 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:25 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:25.871+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:26.196 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:25 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:25.871+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:26.196 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:25 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:25.871+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:26.371 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:26 smithi130 podman[68525]: 2024-04-26 20:51:26.12297721 +0000 UTC m=+5.619124342 container died 765151e63811cf4392fb0119efd954017c69eeebf4293271187dae2cb9984138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T20:51:26.648 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:26.179+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:26.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:26.179+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.086264+0000 front 2024-04-26T20:50:59.086205+0000 (oldest deadline 2024-04-26T20:51:23.786306+0000) 2024-04-26T20:51:26.764 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:26 smithi130 podman[68525]: 2024-04-26 20:51:26.513297935 +0000 UTC m=+6.009445094 container cleanup 765151e63811cf4392fb0119efd954017c69eeebf4293271187dae2cb9984138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-26T20:51:26.765 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:26 smithi130 bash[68525]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5 2024-04-26T20:51:26.765 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:26 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:26.587+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:26.765 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:26 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:26.587+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:26.765 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:26 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:26.587+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:26.766 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:26 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:26.565+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:26.766 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:26 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:26.565+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:27.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:26 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:26.769+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:27.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:26 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:26.769+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:27.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:26 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:26.914+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:27.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:26 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:26.914+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:27.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:26 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:26.914+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:27.121 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:26 smithi130 podman[68542]: 2024-04-26 20:51:26.764774184 +0000 UTC m=+0.638852931 container remove 765151e63811cf4392fb0119efd954017c69eeebf4293271187dae2cb9984138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-26T20:51:27.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:26.743+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:27.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:26.743+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:27.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:26 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:26.743+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:27.436 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:27 smithi130 podman[68634]: 2024-04-26 20:51:27.079530604 +0000 UTC m=+0.017989331 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:51:27.436 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:27 smithi130 podman[68634]: 2024-04-26 20:51:27.190740574 +0000 UTC m=+0.129199289 container create 10eea4c07ce70142810a58de97b148cc230f97d4d37ed6a262492d018c1c8afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T20:51:27.436 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:27 smithi130 podman[68634]: 2024-04-26 20:51:27.436032639 +0000 UTC m=+0.374491353 container init 10eea4c07ce70142810a58de97b148cc230f97d4d37ed6a262492d018c1c8afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T20:51:27.648 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:27 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:27.222+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:27.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:27 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:27.222+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.086264+0000 front 2024-04-26T20:50:59.086205+0000 (oldest deadline 2024-04-26T20:51:23.786306+0000) 2024-04-26T20:51:27.728 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:27 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:27.586+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:27.729 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:27 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:27.586+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:27.729 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:27 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:27.586+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:27.729 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:27 smithi130 podman[68634]: 2024-04-26 20:51:27.439315081 +0000 UTC m=+0.377773796 container start 10eea4c07ce70142810a58de97b148cc230f97d4d37ed6a262492d018c1c8afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-26T20:51:27.729 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:27 smithi130 podman[68634]: 2024-04-26 20:51:27.528321081 +0000 UTC m=+0.466779812 container attach 10eea4c07ce70142810a58de97b148cc230f97d4d37ed6a262492d018c1c8afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-26T20:51:27.729 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:27 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:27.555+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:27.730 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:27 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:27.555+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:28.006 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 26 20:51:27 smithi130 podman[68634]: 2024-04-26 20:51:27.729125487 +0000 UTC m=+0.667584203 container died 10eea4c07ce70142810a58de97b148cc230f97d4d37ed6a262492d018c1c8afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-26T20:51:28.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:27 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:27.785+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:28.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:27 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:27.785+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:28.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:27 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:27.785+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:28.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:27 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:27.799+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:28.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:27 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:27.799+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:28.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:27 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:27.937+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:28.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:27 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:27.937+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:28.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:27 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:27.937+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:28.323 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.5.service' 2024-04-26T20:51:28.402 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:51:28.402 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-26T20:51:28.402 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T20:51:28.402 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.6 2024-04-26T20:51:28.519 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:28.241+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:28.519 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:28.241+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.086264+0000 front 2024-04-26T20:50:59.086205+0000 (oldest deadline 2024-04-26T20:51:23.786306+0000) 2024-04-26T20:51:28.802 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:28 smithi102 systemd[1]: Stopping Ceph osd.6 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:51:28.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:28 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:28.597+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:28.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:28 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:28.597+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:28.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:28 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:28.557+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:28.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:28 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:28.557+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:28.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:28 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:28.557+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:29.149 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:28.957+0000 7f54e7e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:51:29.149 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:28.957+0000 7f54e7e00700 -1 osd.6 97 *** Got signal Terminated *** 2024-04-26T20:51:29.149 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:28.957+0000 7f54e7e00700 -1 osd.6 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:51:29.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:28.800+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:29.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:28.800+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:29.150 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:28 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:28.800+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:29.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:28 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:28.782+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:29.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:28 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:28.782+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:29.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:28 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:28.954+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:29.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:28 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:28.954+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:29.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:28 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:28.954+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:29.648 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:29 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:29.245+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:29.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:29 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:29.245+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.086264+0000 front 2024-04-26T20:50:59.086205+0000 (oldest deadline 2024-04-26T20:51:23.786306+0000) 2024-04-26T20:51:29.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:29 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:29.513+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:29.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:29 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:29.513+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:29.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:29 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:29.513+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:29.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:29 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:29.628+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:29.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:29 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:29.628+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:30.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:29 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:29.757+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:30.027 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:29 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:29.757+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:30.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:29 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:29.941+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:30.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:29 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:29.941+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:30.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:29 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:29.941+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:30.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:29 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:29.833+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:30.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:29 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:29.833+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:30.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:29 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:29.833+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:30.648 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:30.215+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:30.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:30.215+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.086264+0000 front 2024-04-26T20:50:59.086205+0000 (oldest deadline 2024-04-26T20:51:23.786306+0000) 2024-04-26T20:51:30.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:30.215+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:05.488724+0000 front 2024-04-26T20:51:05.488608+0000 (oldest deadline 2024-04-26T20:51:30.188884+0000) 2024-04-26T20:51:30.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:30 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:30.493+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:30.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:30 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:30.493+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:30.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:30 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:30.493+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:30.873 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:30 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:30.594+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:30.873 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:30 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:30.594+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:30.873 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:30 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:30.594+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:31.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:30.859+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:31.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:30.859+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:31.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:30.859+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:31.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:30 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:30.801+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:31.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:30 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:30.801+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:31.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:30 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:30.801+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:31.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:30 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:30.930+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:31.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:30 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:30.930+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:31.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:30 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:30.930+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:31.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:30 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:30.930+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:31.650 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:31 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:31.191+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:31.651 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:31 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:31.191+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.086264+0000 front 2024-04-26T20:50:59.086205+0000 (oldest deadline 2024-04-26T20:51:23.786306+0000) 2024-04-26T20:51:31.651 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:31 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:31.191+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:05.488724+0000 front 2024-04-26T20:51:05.488608+0000 (oldest deadline 2024-04-26T20:51:30.188884+0000) 2024-04-26T20:51:31.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:31 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:31.460+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:31.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:31 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:31.460+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:31.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:31 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:31.460+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:31.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:31 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:31.641+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:31.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:31 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:31.641+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:31.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:31 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:31.641+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:32.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:31 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:31.890+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:32.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:31 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:31.890+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:32.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:31 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:31.890+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:32.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:31 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:31.798+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:32.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:31 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:31.798+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:32.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:31 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:31.798+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:32.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:31 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:31.939+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:32.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:31 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:31.939+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:32.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:31 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:31.939+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:32.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:31 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:31.939+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:32.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:32 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:32.201+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:32.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:32 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:32.201+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.086264+0000 front 2024-04-26T20:50:59.086205+0000 (oldest deadline 2024-04-26T20:51:23.786306+0000) 2024-04-26T20:51:32.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:32 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:32.201+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:05.488724+0000 front 2024-04-26T20:51:05.488608+0000 (oldest deadline 2024-04-26T20:51:30.188884+0000) 2024-04-26T20:51:32.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:32 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:32.456+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:32.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:32 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:32.456+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:32.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:32 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:32.456+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:32.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:32 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:32.679+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:32.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:32 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:32.679+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:32.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:32 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:32.679+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:33.236 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:32 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:32.912+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:33.237 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:32 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:32.912+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:33.237 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:32 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:32.912+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:33.237 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:32 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:32.912+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:33.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:32 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:32.842+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:33.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:32 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:32.842+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:33.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:32 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:32.842+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:33.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:32 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:32.964+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:33.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:32 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:32.964+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:33.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:32 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:32.964+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:33.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:32 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:32.964+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:33.648 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:33.234+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:48.582884+0000 front 2024-04-26T20:50:48.582777+0000 (oldest deadline 2024-04-26T20:51:14.482654+0000) 2024-04-26T20:51:33.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:33.234+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.086264+0000 front 2024-04-26T20:50:59.086205+0000 (oldest deadline 2024-04-26T20:51:23.786306+0000) 2024-04-26T20:51:33.649 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6[56017]: 2024-04-26T20:51:33.234+0000 7f54de600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:05.488724+0000 front 2024-04-26T20:51:05.488608+0000 (oldest deadline 2024-04-26T20:51:30.188884+0000) 2024-04-26T20:51:33.698 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:33 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:33.414+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:33.699 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:33 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:33.414+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:33.699 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:33 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:33.414+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:34.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:33 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:33.698+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:34.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:33 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:33.698+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:34.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:33 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:33.698+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:34.264 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:33.903+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:34.265 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:33.903+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:34.265 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:33.903+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:34.265 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:33 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:33.903+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:34.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:33 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:33.799+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:34.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:33 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:33.799+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:34.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:33 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:33.799+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:34.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:33 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:33.995+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:34.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:33 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:33.995+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:34.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:33 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:33.995+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:34.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:33 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:33.995+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:34.628 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:34 smithi102 podman[86247]: 2024-04-26 20:51:34.396229123 +0000 UTC m=+5.576951293 container died bd0514a4e41b691d2e97cee38c5deae9911658f92561da162b93b74a1d18325e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T20:51:34.733 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:34 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:34.442+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:34.734 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:34 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:34.442+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:34.734 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:34 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:34.442+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:34.881 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:34 smithi102 podman[86247]: 2024-04-26 20:51:34.818545944 +0000 UTC m=+5.999268134 container cleanup bd0514a4e41b691d2e97cee38c5deae9911658f92561da162b93b74a1d18325e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:51:34.882 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:34 smithi102 bash[86247]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6 2024-04-26T20:51:35.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:34 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:34.733+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:35.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:34 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:34.733+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:35.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:34 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:34.733+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:35.148 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:35 smithi102 podman[86268]: 2024-04-26 20:51:35.102923158 +0000 UTC m=+0.693169156 container remove bd0514a4e41b691d2e97cee38c5deae9911658f92561da162b93b74a1d18325e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T20:51:35.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:34 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:34.879+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:35.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:34 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:34.879+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:35.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:34 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:34.879+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:35.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:34 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:34.879+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:35.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:34 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:34.788+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:35.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:34 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:34.788+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:35.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:34 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:34.788+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:35.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:35.007+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:35.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:35.007+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:35.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:35.007+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:35.278 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:35.007+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:35.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:35 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:35.471+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:35.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:35 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:35.471+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:35.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:35 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:35.471+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:35.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:35 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:35.693+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:35.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:35 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:35.693+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:35.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:35 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:35.693+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:36.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:35.757+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:36.027 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:35.757+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:36.027 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:35.757+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:36.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:35.961+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:36.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:35.961+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:36.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:35.961+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:36.028 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:35.961+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:36.148 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:35 smithi102 podman[86365]: 2024-04-26 20:51:35.858725454 +0000 UTC m=+0.042557058 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:51:36.149 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:35 smithi102 podman[86365]: 2024-04-26 20:51:35.978766355 +0000 UTC m=+0.162597995 container create 900fc703d1239c28241f19acb49a0a7b50783fb0dfce83c569e27975ccb1e29e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:51:36.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:35 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:35.841+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:36.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:35 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:35.841+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:36.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:35 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:35.841+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:36.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:35 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:35.841+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:36.407 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:36 smithi102 podman[86365]: 2024-04-26 20:51:36.27458244 +0000 UTC m=+0.458414090 container init 900fc703d1239c28241f19acb49a0a7b50783fb0dfce83c569e27975ccb1e29e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:51:36.407 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:36 smithi102 podman[86365]: 2024-04-26 20:51:36.286378588 +0000 UTC m=+0.470210232 container start 900fc703d1239c28241f19acb49a0a7b50783fb0dfce83c569e27975ccb1e29e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T20:51:36.733 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:36 smithi102 podman[86365]: 2024-04-26 20:51:36.406571218 +0000 UTC m=+0.590402881 container attach 900fc703d1239c28241f19acb49a0a7b50783fb0dfce83c569e27975ccb1e29e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424) 2024-04-26T20:51:36.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:36 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:36.512+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:36.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:36 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:36.512+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:36.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:36 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:36.512+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:36.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:36 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:36.660+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:36.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:36 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:36.660+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:36.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:36 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:36.660+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:37.002 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:36 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:36.773+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:37.002 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:36 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:36.773+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:37.002 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:36 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:36.773+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:37.008 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 26 20:51:36 smithi102 podman[86365]: 2024-04-26 20:51:36.733906327 +0000 UTC m=+0.917737987 container died 900fc703d1239c28241f19acb49a0a7b50783fb0dfce83c569e27975ccb1e29e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T20:51:37.008 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:36 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:36.882+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:37.008 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:36 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:36.882+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:37.008 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:36 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:36.882+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:37.009 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:36 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:36.882+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:37.276 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:37.001+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:37.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:37.001+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:37.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:37.001+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:37.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:37.001+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:37.392 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T20:51:37.502 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:51:37.503 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T20:51:37.503 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-26T20:51:37.503 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.7 2024-04-26T20:51:37.808 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:37 smithi112 systemd[1]: Stopping Ceph osd.7 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:51:37.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:37 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:37.471+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:37.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:37 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:37.471+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:37.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:37 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:37.471+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:37.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:37 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:37.656+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:37.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:37 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:37.656+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:37.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:37 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:37.656+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:38.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:37.992+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:38.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:37.992+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:38.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:37.992+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:38.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:37.992+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:38.278 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:37.808+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:38.278 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:37.808+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:38.278 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:37.808+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:38.278 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:37.836+0000 7f9946fdd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:51:38.278 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:37.836+0000 7f9946fdd700 -1 osd.7 97 *** Got signal Terminated *** 2024-04-26T20:51:38.278 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:37 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:37.836+0000 7f9946fdd700 -1 osd.7 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:51:38.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:37 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:37.899+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:38.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:37 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:37.899+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:38.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:37 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:37.899+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:38.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:37 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:37.899+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:38.706 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:38 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:38.437+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:38.706 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:38 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:38.437+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:38.706 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:38 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:38.437+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:39.026 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:38 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:38.763+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:39.027 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:38 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:38.763+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:39.027 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:38 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:38.764+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:39.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:39 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:39.012+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:39.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:39 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:39.012+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:39.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:39 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:39.012+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:39.027 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:39 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:39.012+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:39.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:38 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:38.705+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:39.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:38 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:38.705+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:39.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:38 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:38.705+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:39.398 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:38 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:38.919+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:39.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:38 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:38.919+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:39.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:38 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:38.919+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:39.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:38 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:38.919+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:39.754 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:39 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:39.457+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:39.755 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:39 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:39.457+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:39.755 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:39 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:39.457+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:40.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:39 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:39.754+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:40.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:39 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:39.754+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:40.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:39 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:39.754+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:40.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:39 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:39.812+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:40.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:39 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:39.812+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:40.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:39 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:39.812+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:40.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:40 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:40.062+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:40.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:40 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:40.062+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:40.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:40 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:40.062+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:40.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:40 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:40.062+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:40.398 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:39 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:39.969+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:40.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:39 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:39.969+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:40.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:39 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:39.969+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:40.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:39 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:39.969+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:40.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:40 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:40.741+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:40.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:40 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:40.742+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:40.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:40 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:40.742+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:40.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:40 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:40.503+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:40.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:40 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:40.503+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:40.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:40 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:40.503+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:41.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:40 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:40.837+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:41.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:40 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:40.837+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:41.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:40 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:40.837+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:41.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:41 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:41.070+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:41.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:41 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:41.070+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:41.278 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:41 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:41.070+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:41.278 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:41 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:41.070+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:41.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:40 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:40.951+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:41.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:40 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:40.951+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:41.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:40 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:40.951+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:41.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:40 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:40.951+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:41.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:41 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:41.739+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:41.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:41 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:41.739+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:41.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:41 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:41.739+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:41.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:41 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:41.739+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:41.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:41 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:41.517+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:41.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:41 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:41.517+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:41.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:41 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:41.517+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:41.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:41 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:41.517+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.570851+0000 front 2024-04-26T20:51:16.570741+0000 (oldest deadline 2024-04-26T20:51:41.270638+0000) 2024-04-26T20:51:42.276 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:41 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:41.867+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:42.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:41 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:41.867+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:42.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:41 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:41.867+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:42.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:41 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:41.867+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.438037+0000 front 2024-04-26T20:51:16.437988+0000 (oldest deadline 2024-04-26T20:51:41.138176+0000) 2024-04-26T20:51:42.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:42 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:42.116+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:42.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:42 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:42.116+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:42.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:42 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:42.116+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:42.278 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:42 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:42.116+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:42.278 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:42 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:42.116+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:42.398 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:41 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:41.940+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:42.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:41 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:41.940+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:42.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:41 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:41.940+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:42.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:41 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:41.940+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:42.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:42 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:42.735+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:42.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:42 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:42.735+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:42.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:42 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:42.735+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:42.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:42 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:42.735+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:42.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:42 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:42.565+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:42.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:42 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:42.565+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:42.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:42 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:42.565+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:42.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:42 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:42.565+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.570851+0000 front 2024-04-26T20:51:16.570741+0000 (oldest deadline 2024-04-26T20:51:41.270638+0000) 2024-04-26T20:51:43.157 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:42 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:42.822+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.233639+0000 front 2024-04-26T20:50:50.233753+0000 (oldest deadline 2024-04-26T20:51:13.733783+0000) 2024-04-26T20:51:43.157 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:42 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:42.822+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.034596+0000 front 2024-04-26T20:50:56.034612+0000 (oldest deadline 2024-04-26T20:51:21.934815+0000) 2024-04-26T20:51:43.157 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:42 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:42.822+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.836340+0000 front 2024-04-26T20:51:07.836008+0000 (oldest deadline 2024-04-26T20:51:30.135927+0000) 2024-04-26T20:51:43.157 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:42 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7[50429]: 2024-04-26T20:51:42.822+0000 7f99435d1700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.438037+0000 front 2024-04-26T20:51:16.437988+0000 (oldest deadline 2024-04-26T20:51:41.138176+0000) 2024-04-26T20:51:43.398 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:42 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:42.906+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:43.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:42 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:42.906+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:43.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:42 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:42.906+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:43.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:42 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:42.906+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:43.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:42 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:42.906+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:43.526 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:43 smithi112 podman[67903]: 2024-04-26 20:51:43.329550114 +0000 UTC m=+5.623750278 container died b77c7bb631459c2bbd88454cbf3b2ca87df23d24494a5ae1022ea0cb70ec2e6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:51:43.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:43 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:43.156+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:43.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:43 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:43.156+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:43.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:43 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:43.156+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:43.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:43 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:43.156+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:43.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:43 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:43.156+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:43.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:43 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:43.705+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:43.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:43 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:43.705+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:43.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:43 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:43.705+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:43.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:43 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:43.705+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:43.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:43 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:43.552+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:43.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:43 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:43.552+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:43.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:43 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:43.552+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:43.874 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:43 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:43.552+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.570851+0000 front 2024-04-26T20:51:16.570741+0000 (oldest deadline 2024-04-26T20:51:41.270638+0000) 2024-04-26T20:51:43.963 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:43 smithi112 podman[67903]: 2024-04-26 20:51:43.702966239 +0000 UTC m=+5.997166412 container cleanup b77c7bb631459c2bbd88454cbf3b2ca87df23d24494a5ae1022ea0cb70ec2e6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T20:51:43.963 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:43 smithi112 bash[67903]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7 2024-04-26T20:51:44.277 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:43 smithi112 podman[67921]: 2024-04-26 20:51:43.962511773 +0000 UTC m=+0.633596504 container remove b77c7bb631459c2bbd88454cbf3b2ca87df23d24494a5ae1022ea0cb70ec2e6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:51:44.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:44 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:44.146+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:44.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:44 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:44.146+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:44.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:44 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:44.146+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:44.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:44 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:44.146+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:44.278 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:44 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:44.146+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:44.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:43 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:43.937+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:44.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:43 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:43.937+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:44.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:43 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:43.937+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:44.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:43 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:43.937+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:44.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:43 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:43.937+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:44.624 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:44 smithi112 podman[68011]: 2024-04-26 20:51:44.226239326 +0000 UTC m=+0.017728316 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:51:44.624 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:44 smithi112 podman[68011]: 2024-04-26 20:51:44.355386572 +0000 UTC m=+0.146875580 container create 3d12e7008e7fbf00ddd8e8612e61c56bb11003df9ed733760f1bd31e508931c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:51:44.624 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:44 smithi112 podman[68011]: 2024-04-26 20:51:44.624497956 +0000 UTC m=+0.415986919 container init 3d12e7008e7fbf00ddd8e8612e61c56bb11003df9ed733760f1bd31e508931c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-26T20:51:44.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:44 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:44.566+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:44.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:44 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:44.566+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:44.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:44 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:44.566+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:44.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:44 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:44.566+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.570851+0000 front 2024-04-26T20:51:16.570741+0000 (oldest deadline 2024-04-26T20:51:41.270638+0000) 2024-04-26T20:51:44.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:44 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:44.732+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:44.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:44 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:44.732+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:44.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:44 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:44.732+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:44.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:44 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:44.732+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:44.924 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:44 smithi112 podman[68011]: 2024-04-26 20:51:44.627882677 +0000 UTC m=+0.419371640 container start 3d12e7008e7fbf00ddd8e8612e61c56bb11003df9ed733760f1bd31e508931c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T20:51:44.924 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:44 smithi112 podman[68011]: 2024-04-26 20:51:44.707882542 +0000 UTC m=+0.499371521 container attach 3d12e7008e7fbf00ddd8e8612e61c56bb11003df9ed733760f1bd31e508931c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-26T20:51:45.211 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 26 20:51:44 smithi112 podman[68011]: 2024-04-26 20:51:44.924838628 +0000 UTC m=+0.716327601 container died 3d12e7008e7fbf00ddd8e8612e61c56bb11003df9ed733760f1bd31e508931c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T20:51:45.211 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:45 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:45.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:45.211 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:45 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:45.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:45.212 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:45 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:45.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:45.212 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:45 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:45.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:45.212 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:45 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:45.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:45.309 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:44 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:44.934+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:45.309 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:44 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:44.934+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:45.310 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:44 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:44.934+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:45.310 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:44 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:44.934+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:45.310 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:44 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:44.934+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:45.535 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.7.service' 2024-04-26T20:51:45.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:51:45.573 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-26T20:51:45.573 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-26T20:51:45.573 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.8 2024-04-26T20:51:45.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:45.720+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:45.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:45.720+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:45.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:45.720+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:45.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:45.720+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:45.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:45.566+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:45.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:45.566+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:45.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:45.566+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:45.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:45.566+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.570851+0000 front 2024-04-26T20:51:16.570741+0000 (oldest deadline 2024-04-26T20:51:41.270638+0000) 2024-04-26T20:51:45.873 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:45 smithi130 systemd[1]: Stopping Ceph osd.8 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:51:46.371 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:45.907+0000 7fb478f4a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:51:46.383 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:45.907+0000 7fb478f4a700 -1 osd.8 97 *** Got signal Terminated *** 2024-04-26T20:51:46.383 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:45 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:45.907+0000 7fb478f4a700 -1 osd.8 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:51:46.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:45 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:45.953+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:46.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:45 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:45.953+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:46.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:45 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:45.953+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:46.400 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:45 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:45.953+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:46.400 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:45 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:45.953+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:46.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:46 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:46.158+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:46.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:46 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:46.158+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:46.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:46 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:46.158+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:46.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:46 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:46.158+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:46.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:46 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:46.158+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:46.805 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:46 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:46.546+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:46.805 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:46 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:46.546+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:46.806 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:46 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:46.546+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:46.806 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:46 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:46.546+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.570851+0000 front 2024-04-26T20:51:16.570741+0000 (oldest deadline 2024-04-26T20:51:41.270638+0000) 2024-04-26T20:51:46.806 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:46 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:46.760+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:46.806 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:46 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:46.760+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:46.806 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:46 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:46.760+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:46.806 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:46 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:46.760+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:47.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:46 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:46.919+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:47.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:46 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:46.919+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:47.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:46 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:46.919+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:47.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:46 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:46.919+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:47.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:46 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:46.919+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:47.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:47 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:47.118+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:47.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:47 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:47.118+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:47.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:47 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:47.118+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:47.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:47 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:47.118+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:47.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:47 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:47.118+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:47.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:47 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:47.575+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:47.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:47 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:47.575+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:47.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:47 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:47.575+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:47.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:47 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:47.575+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.570851+0000 front 2024-04-26T20:51:16.570741+0000 (oldest deadline 2024-04-26T20:51:41.270638+0000) 2024-04-26T20:51:47.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:47 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:47.790+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:47.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:47 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:47.790+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:47.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:47 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:47.790+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:47.872 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:47 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:47.790+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:48.398 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:47 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:47.965+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:48.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:47 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:47.965+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:48.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:47 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:47.965+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:48.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:47 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:47.965+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:48.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:47 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:47.965+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:48.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:48 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:48.107+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:48.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:48 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:48.107+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:48.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:48 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:48.107+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:48.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:48 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:48.107+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:48.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:48 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:48.107+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:48.810 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:48.538+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:48.811 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:48.538+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:48.811 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:48.538+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:48.811 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:48.538+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.570851+0000 front 2024-04-26T20:51:16.570741+0000 (oldest deadline 2024-04-26T20:51:41.270638+0000) 2024-04-26T20:51:49.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:48.809+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:49.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:48.809+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:49.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:48.809+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:49.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:48.809+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:49.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:48 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:48.809+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:49.398 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:48 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:48.971+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:49.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:48 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:48.971+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:49.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:48 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:48.971+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:49.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:48 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:48.971+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:49.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:48 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:48.971+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:49.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:49 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:49.110+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:49.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:49 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:49.110+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:49.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:49 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:49.110+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:49.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:49 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:49.110+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:49.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:49 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:49.110+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:49.839 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:49.503+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:49.839 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:49.503+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:49.839 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:49.503+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:49.839 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:49.503+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.570851+0000 front 2024-04-26T20:51:16.570741+0000 (oldest deadline 2024-04-26T20:51:41.270638+0000) 2024-04-26T20:51:50.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:49.838+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:50.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:49.838+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:50.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:49.838+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:50.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:49.838+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:50.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:49 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:49.838+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:50.398 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:49 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:49.975+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:50.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:49 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:49.975+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:50.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:49 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:49.975+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:50.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:49 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:49.975+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:50.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:49 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:49.975+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:50.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:49 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:49.975+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:23.985736+0000 front 2024-04-26T20:51:23.985980+0000 (oldest deadline 2024-04-26T20:51:49.285601+0000) 2024-04-26T20:51:50.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:50 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:50.120+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:50.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:50 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:50.120+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:50.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:50 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:50.120+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:50.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:50 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:50.120+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:50.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:50 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:50.120+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:50.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:50 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:50.120+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:51:50.843 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:50.500+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:38.264194+0000 front 2024-04-26T20:50:38.264290+0000 (oldest deadline 2024-04-26T20:51:04.164061+0000) 2024-04-26T20:51:50.843 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:50.500+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.565603+0000 front 2024-04-26T20:50:50.565690+0000 (oldest deadline 2024-04-26T20:51:16.465543+0000) 2024-04-26T20:51:50.843 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:50.500+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.767450+0000 front 2024-04-26T20:50:59.767572+0000 (oldest deadline 2024-04-26T20:51:20.867562+0000) 2024-04-26T20:51:50.844 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8[51328]: 2024-04-26T20:51:50.500+0000 7fb47553e700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.570851+0000 front 2024-04-26T20:51:16.570741+0000 (oldest deadline 2024-04-26T20:51:41.270638+0000) 2024-04-26T20:51:51.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:50.842+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:51.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:50.842+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:51.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:50.842+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:51.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:50.842+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:51.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:50 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:50.842+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:51.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:51.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:51.400 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:51.400 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:51.400 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:51.400 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:23.985736+0000 front 2024-04-26T20:51:23.985980+0000 (oldest deadline 2024-04-26T20:51:49.285601+0000) 2024-04-26T20:51:51.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:51 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:51.072+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:51.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:51 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:51.072+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:51.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:51 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:51.073+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:51.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:51 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:51.073+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:51.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:51 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:51.073+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:51.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:51 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:51.073+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:51:51.621 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:51 smithi130 podman[68739]: 2024-04-26 20:51:51.327836949 +0000 UTC m=+5.558521912 container died 0099655d5f37caed770ab80c21c98141a74c596aaf45e87f458ebed16944d5fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-26T20:51:51.986 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:51 smithi130 podman[68739]: 2024-04-26 20:51:51.743482347 +0000 UTC m=+5.974167354 container cleanup 0099655d5f37caed770ab80c21c98141a74c596aaf45e87f458ebed16944d5fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:51:51.987 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:51 smithi130 bash[68739]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8 2024-04-26T20:51:51.987 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:51 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:51.871+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:51.987 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:51 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:51.871+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:51.987 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:51 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:51.871+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:51.987 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:51 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:51.871+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:51.987 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:51 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:51.871+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:52.371 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:51 smithi130 podman[68767]: 2024-04-26 20:51:51.986576173 +0000 UTC m=+0.653489253 container remove 0099655d5f37caed770ab80c21c98141a74c596aaf45e87f458ebed16944d5fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T20:51:52.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.982+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:52.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.982+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:52.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.982+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:52.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.982+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:52.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.982+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:52.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:51 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:51.982+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:23.985736+0000 front 2024-04-26T20:51:23.985980+0000 (oldest deadline 2024-04-26T20:51:49.285601+0000) 2024-04-26T20:51:52.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:52 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:52.029+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:52.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:52 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:52.029+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:52.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:52 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:52.029+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:52.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:52 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:52.029+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:52.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:52 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:52.029+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:52.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:52 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:52.029+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:51:52.690 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:52 smithi130 podman[68858]: 2024-04-26 20:51:52.328067189 +0000 UTC m=+0.018073598 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:51:52.690 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:52 smithi130 podman[68858]: 2024-04-26 20:51:52.437436901 +0000 UTC m=+0.127443296 container create 7cbb321d5852c8d7c8cbeebef7eb561ac832995bb5e8a04e55369c7db7c5b21a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T20:51:52.690 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:52 smithi130 podman[68858]: 2024-04-26 20:51:52.690260864 +0000 UTC m=+0.380267260 container init 7cbb321d5852c8d7c8cbeebef7eb561ac832995bb5e8a04e55369c7db7c5b21a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T20:51:52.985 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:52 smithi130 podman[68858]: 2024-04-26 20:51:52.693466588 +0000 UTC m=+0.383472985 container start 7cbb321d5852c8d7c8cbeebef7eb561ac832995bb5e8a04e55369c7db7c5b21a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-26T20:51:52.985 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:52 smithi130 podman[68858]: 2024-04-26 20:51:52.790356894 +0000 UTC m=+0.480363302 container attach 7cbb321d5852c8d7c8cbeebef7eb561ac832995bb5e8a04e55369c7db7c5b21a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T20:51:52.985 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:52 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:52.833+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:52.985 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:52 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:52.833+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:52.985 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:52 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:52.833+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:52.986 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:52 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:52.833+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:52.986 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:52 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:52.833+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:53.343 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 26 20:51:52 smithi130 podman[68858]: 2024-04-26 20:51:52.984818156 +0000 UTC m=+0.674824556 container died 7cbb321d5852c8d7c8cbeebef7eb561ac832995bb5e8a04e55369c7db7c5b21a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:51:53.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.004+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:53.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.004+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:53.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.004+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:53.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.004+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:53.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.004+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:53.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.004+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:23.985736+0000 front 2024-04-26T20:51:23.985980+0000 (oldest deadline 2024-04-26T20:51:49.285601+0000) 2024-04-26T20:51:53.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:53 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:53.049+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:53.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:53 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:53.049+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:53.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:53 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:53.049+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:53.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:53 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:53.049+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:53.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:53 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:53.049+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:53.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:53 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:53.049+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:51:53.687 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.8.service' 2024-04-26T20:51:53.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:51:53.726 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-26T20:51:53.726 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-26T20:51:53.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.9 2024-04-26T20:51:54.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:53.805+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:54.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:53.805+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:54.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:53.805+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:54.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:53.805+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:54.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:53.805+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:54.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 systemd[1]: Stopping Ceph osd.9 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:51:54.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.968+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:54.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.968+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:54.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.968+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:54.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.968+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:54.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.968+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:54.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:53 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:53.968+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:23.985736+0000 front 2024-04-26T20:51:23.985980+0000 (oldest deadline 2024-04-26T20:51:49.285601+0000) 2024-04-26T20:51:54.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:54 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:54.072+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:54.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:54 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:54.072+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:54.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:54 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:54.072+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:54.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:54 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:54.072+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:54.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:54 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:54.072+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:54.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:54 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:54.072+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:51:54.648 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:54.261+0000 7feb6bc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:51:54.649 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:54.261+0000 7feb6bc00700 -1 osd.9 97 *** Got signal Terminated *** 2024-04-26T20:51:54.649 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:54.261+0000 7feb6bc00700 -1 osd.9 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:51:55.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:54 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:54.805+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:55.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:54 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:54.805+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:55.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:54 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:54.805+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:55.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:54 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:54.805+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:55.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:54 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:54.805+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:55.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:54.958+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:55.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:54.958+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:55.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:54.958+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:55.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:54.958+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:55.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:54.958+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:55.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:54 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:54.958+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:23.985736+0000 front 2024-04-26T20:51:23.985980+0000 (oldest deadline 2024-04-26T20:51:49.285601+0000) 2024-04-26T20:51:55.428 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:55.108+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:55.428 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:55.108+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:55.428 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:55.108+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:55.428 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:55.108+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:55.429 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:55.108+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:55.429 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:55 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:55.108+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:51:56.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:55 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:55.819+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:56.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:55 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:55.819+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:56.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:55 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:55.819+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:56.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:55 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:55.819+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:56.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:55 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:55.819+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:56.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:55 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:55.819+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:51:56.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.001+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:56.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.001+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:56.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.001+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:56.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.001+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:56.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.001+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:56.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.001+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:23.985736+0000 front 2024-04-26T20:51:23.985980+0000 (oldest deadline 2024-04-26T20:51:49.285601+0000) 2024-04-26T20:51:56.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:56 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:56.155+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:56.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:56 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:56.155+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:56.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:56 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:56.155+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:56.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:56 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:56.155+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:56.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:56 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:56.155+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:56.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:56 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:56.155+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:51:57.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:56 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:56.785+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:57.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:56 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:56.785+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:57.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:56 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:56.785+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:57.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:56 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:56.785+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:57.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:56 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:56.785+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:57.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:56 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:56.785+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:51:57.398 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.981+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:57.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.981+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:57.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.981+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:57.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.981+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:57.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.981+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:57.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:56 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:56.981+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:23.985736+0000 front 2024-04-26T20:51:23.985980+0000 (oldest deadline 2024-04-26T20:51:49.285601+0000) 2024-04-26T20:51:57.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:57 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:57.144+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:57.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:57 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:57.144+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:57.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:57 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:57.144+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:57.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:57 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:57.144+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:57.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:57 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:57.144+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:57.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:57 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:57.144+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:51:57.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:57 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:57.144+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:51:58.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:57 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:57.742+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:58.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:57 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:57.742+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:58.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:57 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:57.742+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:58.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:57 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:57.742+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:58.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:57 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:57.742+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:58.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:57 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:57.742+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:51:58.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:58.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:58.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:58.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:58.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:58.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.019+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:23.985736+0000 front 2024-04-26T20:51:23.985980+0000 (oldest deadline 2024-04-26T20:51:49.285601+0000) 2024-04-26T20:51:58.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:58 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:58.168+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:58.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:58 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:58.169+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:58.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:58 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:58.169+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:58.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:58 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:58.169+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:58.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:58 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:58.169+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:58.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:58 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:58.169+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:51:58.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:58 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:58.169+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:51:59.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:58 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:58.725+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:51:59.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:58 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:58.725+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:51:59.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:58 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:58.725+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:51:59.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:58 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:58.725+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:51:59.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:58 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:58.725+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:51:59.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:58 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:58.725+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:51:59.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.990+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.069520+0000 front 2024-04-26T20:50:41.069925+0000 (oldest deadline 2024-04-26T20:51:03.870860+0000) 2024-04-26T20:51:59.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.990+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:50.675734+0000 front 2024-04-26T20:50:50.675608+0000 (oldest deadline 2024-04-26T20:51:14.175841+0000) 2024-04-26T20:51:59.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.990+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:55.878347+0000 front 2024-04-26T20:50:55.878539+0000 (oldest deadline 2024-04-26T20:51:21.778582+0000) 2024-04-26T20:51:59.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.990+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:07.080634+0000 front 2024-04-26T20:51:07.080735+0000 (oldest deadline 2024-04-26T20:51:32.380855+0000) 2024-04-26T20:51:59.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.990+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:16.983768+0000 front 2024-04-26T20:51:16.983974+0000 (oldest deadline 2024-04-26T20:51:42.883734+0000) 2024-04-26T20:51:59.399 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:58 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9[62572]: 2024-04-26T20:51:58.990+0000 7feb67400700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:23.985736+0000 front 2024-04-26T20:51:23.985980+0000 (oldest deadline 2024-04-26T20:51:49.285601+0000) 2024-04-26T20:51:59.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:59 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:59.172+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:51:59.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:59 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:59.172+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:51:59.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:59 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:59.172+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:51:59.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:59 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:59.172+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:51:59.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:59 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:59.172+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:51:59.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:59 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:59.172+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:51:59.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:51:59 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:51:59.172+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:51:59.869 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:51:59 smithi102 podman[86472]: 2024-04-26 20:51:59.609983648 +0000 UTC m=+5.484883111 container died ae79d4304d9259f835e8e027fa180b24ae7d4b80ed7c1f43c8c3e634a7907b18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T20:52:00.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:59 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:59.694+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:00.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:59 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:59.694+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:00.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:59 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:59.694+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:00.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:59 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:59.694+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:00.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:59 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:59.694+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:00.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:51:59 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:51:59.694+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:00.148 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:52:00 smithi102 podman[86472]: 2024-04-26 20:52:00.020518205 +0000 UTC m=+5.895417631 container cleanup ae79d4304d9259f835e8e027fa180b24ae7d4b80ed7c1f43c8c3e634a7907b18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T20:52:00.149 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:52:00 smithi102 bash[86472]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9 2024-04-26T20:52:00.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:00 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:00.164+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:52:00.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:00 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:00.164+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:52:00.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:00 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:00.164+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:52:00.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:00 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:00.164+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:52:00.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:00 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:00.164+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:52:00.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:00 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:00.164+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:52:00.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:00 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:00.164+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:52:00.648 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:52:00 smithi102 podman[86490]: 2024-04-26 20:52:00.204806638 +0000 UTC m=+0.579876841 container remove ae79d4304d9259f835e8e027fa180b24ae7d4b80ed7c1f43c8c3e634a7907b18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux ) 2024-04-26T20:52:01.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:00 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:00.668+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:01.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:00 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:00.668+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:01.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:00 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:00.668+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:01.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:00 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:00.668+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:01.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:00 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:00.668+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:01.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:00 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:00.668+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:01.335 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:52:01 smithi102 podman[86575]: 2024-04-26 20:52:00.958480086 +0000 UTC m=+0.033811920 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:52:01.335 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:52:01 smithi102 podman[86575]: 2024-04-26 20:52:01.055544466 +0000 UTC m=+0.130876320 container create 825cf61f4c52b091323aab69b4b2e0ce304242b608ad5f827d5483c445c14206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:52:01.336 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:52:01 smithi102 podman[86575]: 2024-04-26 20:52:01.334359769 +0000 UTC m=+0.409691662 container init 825cf61f4c52b091323aab69b4b2e0ce304242b608ad5f827d5483c445c14206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:52:01.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:01 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:01.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:52:01.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:01 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:01.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:52:01.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:01 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:01.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:52:01.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:01 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:01.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:52:01.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:01 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:01.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:52:01.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:01 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:01.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:52:01.528 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:01 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:01.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:52:01.648 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:52:01 smithi102 podman[86575]: 2024-04-26 20:52:01.346688341 +0000 UTC m=+0.422020179 container start 825cf61f4c52b091323aab69b4b2e0ce304242b608ad5f827d5483c445c14206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:52:01.648 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:52:01 smithi102 podman[86575]: 2024-04-26 20:52:01.424866185 +0000 UTC m=+0.500198102 container attach 825cf61f4c52b091323aab69b4b2e0ce304242b608ad5f827d5483c445c14206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T20:52:02.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:01 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:01.698+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:02.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:01 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:01.698+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:02.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:01 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:01.698+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:02.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:01 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:01.698+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:02.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:01 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:01.698+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:02.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:01 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:01.698+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:02.147 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 26 20:52:01 smithi102 podman[86575]: 2024-04-26 20:52:01.799182147 +0000 UTC m=+0.874513983 container died 825cf61f4c52b091323aab69b4b2e0ce304242b608ad5f827d5483c445c14206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-26T20:52:02.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.9.service' 2024-04-26T20:52:02.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:02.189+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:52:02.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:02.189+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:52:02.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:02.189+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:52:02.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:02.189+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:52:02.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:02.189+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:52:02.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:02.189+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:52:02.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:02.189+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:52:02.610 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:02.610 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-26T20:52:02.610 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-26T20:52:02.610 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.10 2024-04-26T20:52:02.923 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 systemd[1]: Stopping Ceph osd.10 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:03.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:02 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:02.732+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:03.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:02 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:02.732+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:03.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:02 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:02.732+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:03.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:02 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:02.732+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:03.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:02 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:02.732+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:03.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:02 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:02.732+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:03.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:02.923+0000 7f1f031f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:03.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:02.923+0000 7f1f031f2700 -1 osd.10 97 *** Got signal Terminated *** 2024-04-26T20:52:03.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:02 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:02.923+0000 7f1f031f2700 -1 osd.10 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:52:03.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:03 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:03.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:52:03.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:03 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:03.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:52:03.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:03 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:03.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:52:03.277 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:03 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:03.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:52:03.278 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:03 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:03.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:52:03.278 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:03 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:03.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:52:03.278 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:03 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:03.170+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:52:04.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:03 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:03.734+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:04.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:03 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:03.734+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:04.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:03 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:03.734+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:04.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:03 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:03.734+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:04.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:03 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:03.734+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:04.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:03 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:03.734+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:04.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:04 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:04.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:52:04.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:04 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:04.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:52:04.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:04 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:04.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:52:04.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:04 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:04.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:52:04.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:04 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:04.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:52:04.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:04 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:04.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:52:04.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:04 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:04.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:52:05.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:04 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:04.750+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:05.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:04 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:04.750+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:05.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:04 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:04.750+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:05.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:04 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:04.750+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:05.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:04 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:04.750+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:05.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:04 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:04.750+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:05.482 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:05 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:05.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:52:05.482 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:05 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:05.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:52:05.482 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:05 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:05.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:52:05.482 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:05 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:05.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:52:05.482 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:05 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:05.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:52:05.483 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:05 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:05.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:52:05.483 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:05 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:05.165+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:52:06.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:05.741+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:06.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:05.741+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:06.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:05.741+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:06.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:05.741+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:06.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:05.741+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:06.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:05.741+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:06.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:05.741+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:06.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:06 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:06.200+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:52:06.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:06 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:06.200+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:52:06.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:06 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:06.200+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:52:06.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:06 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:06.200+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:52:06.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:06 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:06.200+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:52:06.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:06 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:06.200+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:52:06.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:06 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:06.200+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:52:07.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:06 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:06.753+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:07.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:06 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:06.753+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:07.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:06 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:06.753+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:07.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:06 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:06.753+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:07.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:06 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:06.753+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:07.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:06 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:06.753+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:07.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:06 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:06.753+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:07.526 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:07 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:07.151+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:37.489800+0000 front 2024-04-26T20:50:37.489519+0000 (oldest deadline 2024-04-26T20:51:03.889868+0000) 2024-04-26T20:52:07.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:07 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:07.151+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-26T20:50:47.891237+0000 front 2024-04-26T20:50:47.891287+0000 (oldest deadline 2024-04-26T20:51:13.791515+0000) 2024-04-26T20:52:07.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:07 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:07.151+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:59.692849+0000 front 2024-04-26T20:50:59.692721+0000 (oldest deadline 2024-04-26T20:51:22.592690+0000) 2024-04-26T20:52:07.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:07 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:07.151+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:08.394597+0000 front 2024-04-26T20:51:08.394462+0000 (oldest deadline 2024-04-26T20:51:30.694302+0000) 2024-04-26T20:52:07.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:07 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:07.151+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:15.295742+0000 front 2024-04-26T20:51:15.295782+0000 (oldest deadline 2024-04-26T20:51:41.195862+0000) 2024-04-26T20:52:07.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:07 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:07.151+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:25.296924+0000 front 2024-04-26T20:51:25.296806+0000 (oldest deadline 2024-04-26T20:51:49.396631+0000) 2024-04-26T20:52:07.527 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:07 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-26T20:52:07.151+0000 7f1eff7e6700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:29.397332+0000 front 2024-04-26T20:51:29.397463+0000 (oldest deadline 2024-04-26T20:51:56.997549+0000) 2024-04-26T20:52:08.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:07 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:07.768+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:08.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:07 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:07.768+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:08.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:07 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:07.768+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:08.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:07 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:07.768+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:08.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:07 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:07.768+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:08.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:07 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:07.768+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:08.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:07 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:07.768+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:08.607 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:08 smithi112 podman[68126]: 2024-04-26 20:52:08.326544462 +0000 UTC m=+5.516958885 container died 138d10015cac80a67fed1d4b3b803e0fe8d40cb857594de6e979dd28f61d674e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T20:52:08.957 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:08 smithi112 podman[68126]: 2024-04-26 20:52:08.689572311 +0000 UTC m=+5.879986743 container cleanup 138d10015cac80a67fed1d4b3b803e0fe8d40cb857594de6e979dd28f61d674e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T20:52:08.957 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:08 smithi112 bash[68126]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10 2024-04-26T20:52:09.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:08.813+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:09.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:08.813+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:09.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:08.813+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:09.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:08.813+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:09.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:08.813+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:09.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:08.813+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:09.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:08 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:08.813+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:09.276 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:08 smithi112 podman[68144]: 2024-04-26 20:52:08.957444884 +0000 UTC m=+0.623619987 container remove 138d10015cac80a67fed1d4b3b803e0fe8d40cb857594de6e979dd28f61d674e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-26T20:52:09.560 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:09 smithi112 podman[68240]: 2024-04-26 20:52:09.213000357 +0000 UTC m=+0.017570554 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:52:09.561 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:09 smithi112 podman[68240]: 2024-04-26 20:52:09.316623293 +0000 UTC m=+0.121193482 container create a5a339ecf0c081c1f88446d85b93a63d610d90916d68508dd8ec7f6cda3443c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T20:52:09.850 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:09 smithi112 podman[68240]: 2024-04-26 20:52:09.560773832 +0000 UTC m=+0.365344019 container init a5a339ecf0c081c1f88446d85b93a63d610d90916d68508dd8ec7f6cda3443c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T20:52:09.850 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:09 smithi112 podman[68240]: 2024-04-26 20:52:09.563822673 +0000 UTC m=+0.368392860 container start a5a339ecf0c081c1f88446d85b93a63d610d90916d68508dd8ec7f6cda3443c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-26T20:52:09.850 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:09 smithi112 podman[68240]: 2024-04-26 20:52:09.652323689 +0000 UTC m=+0.456893884 container attach a5a339ecf0c081c1f88446d85b93a63d610d90916d68508dd8ec7f6cda3443c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T20:52:10.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:09.772+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:10.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:09.772+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:10.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:09.772+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:10.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:09.772+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:10.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:09.772+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:10.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:09.772+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:10.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:09 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:09.772+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:10.146 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 26 20:52:09 smithi112 podman[68240]: 2024-04-26 20:52:09.850374728 +0000 UTC m=+0.654944905 container died a5a339ecf0c081c1f88446d85b93a63d610d90916d68508dd8ec7f6cda3443c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T20:52:10.434 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.10.service' 2024-04-26T20:52:10.472 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:10.473 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-26T20:52:10.473 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-26T20:52:10.473 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.11 2024-04-26T20:52:10.784 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 systemd[1]: Stopping Ceph osd.11 for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:11.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:10.783+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:11.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:10.783+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:11.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:10.783+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:11.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:10.783+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:11.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:10.783+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:11.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:10.783+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:11.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:10.783+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:11.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:10.838+0000 7f71bd201700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:11.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:10.838+0000 7f71bd201700 -1 osd.11 97 *** Got signal Terminated *** 2024-04-26T20:52:11.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:10 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:10.838+0000 7f71bd201700 -1 osd.11 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T20:52:12.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:11 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:11.814+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:12.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:11 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:11.814+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:12.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:11 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:11.814+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:12.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:11 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:11.814+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:12.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:11 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:11.814+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:12.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:11 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:11.814+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:12.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:11 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:11.814+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:13.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:12 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:12.859+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:13.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:12 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:12.859+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:13.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:12 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:12.859+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:13.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:12 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:12.859+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:13.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:12 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:12.859+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:13.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:12 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:12.859+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:13.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:12 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:12.859+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:13.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:12 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:12.859+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6820 osd.8 since back 2024-04-26T20:51:48.141195+0000 front 2024-04-26T20:51:48.141177+0000 (oldest deadline 2024-04-26T20:52:12.241249+0000) 2024-04-26T20:52:14.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:13.826+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:14.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:13.826+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:14.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:13.826+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:14.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:13.826+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:14.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:13.826+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:14.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:13.826+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:14.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:13.826+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:14.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:13 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:13.826+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6820 osd.8 since back 2024-04-26T20:51:48.141195+0000 front 2024-04-26T20:51:48.141177+0000 (oldest deadline 2024-04-26T20:52:12.241249+0000) 2024-04-26T20:52:15.121 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:14.792+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:15.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:14.792+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:15.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:14.792+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:15.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:14.792+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:15.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:14.792+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:15.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:14.792+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:15.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:14.792+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:15.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:14 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:14.792+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6820 osd.8 since back 2024-04-26T20:51:48.141195+0000 front 2024-04-26T20:51:48.141177+0000 (oldest deadline 2024-04-26T20:52:12.241249+0000) 2024-04-26T20:52:16.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:15.771+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-26T20:50:41.930503+0000 front 2024-04-26T20:50:41.930354+0000 (oldest deadline 2024-04-26T20:51:06.030317+0000) 2024-04-26T20:52:16.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:15.771+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-26T20:50:56.532279+0000 front 2024-04-26T20:50:56.532339+0000 (oldest deadline 2024-04-26T20:51:24.132931+0000) 2024-04-26T20:52:16.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:15.771+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-26T20:51:04.133768+0000 front 2024-04-26T20:51:04.133845+0000 (oldest deadline 2024-04-26T20:51:30.033545+0000) 2024-04-26T20:52:16.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:15.771+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-26T20:51:17.035193+0000 front 2024-04-26T20:51:17.035200+0000 (oldest deadline 2024-04-26T20:51:41.735194+0000) 2024-04-26T20:52:16.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:15.771+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-26T20:51:21.735532+0000 front 2024-04-26T20:51:21.735589+0000 (oldest deadline 2024-04-26T20:51:48.136255+0000) 2024-04-26T20:52:16.122 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:15.771+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-26T20:51:32.038399+0000 front 2024-04-26T20:51:32.038518+0000 (oldest deadline 2024-04-26T20:51:54.938185+0000) 2024-04-26T20:52:16.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:15.771+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-26T20:51:39.539774+0000 front 2024-04-26T20:51:39.539752+0000 (oldest deadline 2024-04-26T20:52:04.839634+0000) 2024-04-26T20:52:16.123 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:15 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11[57936]: 2024-04-26T20:52:15.771+0000 7f71b57ed700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.130:6820 osd.8 since back 2024-04-26T20:51:48.141195+0000 front 2024-04-26T20:51:48.141177+0000 (oldest deadline 2024-04-26T20:52:12.241249+0000) 2024-04-26T20:52:16.514 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:16 smithi130 podman[68963]: 2024-04-26 20:52:16.243891152 +0000 UTC m=+5.581181290 container died a60b3e00262759d591624a1ba8723c09dcbddf68bfedbef3ff68ae92cb3671c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T20:52:16.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:16 smithi130 podman[68963]: 2024-04-26 20:52:16.613910546 +0000 UTC m=+5.951200724 container cleanup a60b3e00262759d591624a1ba8723c09dcbddf68bfedbef3ff68ae92cb3671c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:52:16.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:16 smithi130 bash[68963]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11 2024-04-26T20:52:17.230 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:16 smithi130 podman[68980]: 2024-04-26 20:52:16.881466909 +0000 UTC m=+0.637098362 container remove a60b3e00262759d591624a1ba8723c09dcbddf68bfedbef3ff68ae92cb3671c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True) 2024-04-26T20:52:17.501 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:17 smithi130 podman[69072]: 2024-04-26 20:52:17.133360411 +0000 UTC m=+0.017083305 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T20:52:17.501 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:17 smithi130 podman[69072]: 2024-04-26 20:52:17.232250612 +0000 UTC m=+0.115973527 container create d1407a32988f18ccea911c17acbba6fbfe39417cfadb5bc9fa79e640d4ed9be3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T20:52:17.501 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:17 smithi130 podman[69072]: 2024-04-26 20:52:17.434769707 +0000 UTC m=+0.318492579 container init d1407a32988f18ccea911c17acbba6fbfe39417cfadb5bc9fa79e640d4ed9be3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T20:52:17.501 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:17 smithi130 podman[69072]: 2024-04-26 20:52:17.437748804 +0000 UTC m=+0.321471684 container start d1407a32988f18ccea911c17acbba6fbfe39417cfadb5bc9fa79e640d4ed9be3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T20:52:17.502 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:17 smithi130 podman[69072]: 2024-04-26 20:52:17.501186714 +0000 UTC m=+0.384909600 container attach d1407a32988f18ccea911c17acbba6fbfe39417cfadb5bc9fa79e640d4ed9be3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:52:17.871 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 26 20:52:17 smithi130 podman[69072]: 2024-04-26 20:52:17.727144068 +0000 UTC m=+0.610866988 container died d1407a32988f18ccea911c17acbba6fbfe39417cfadb5bc9fa79e640d4ed9be3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T20:52:18.312 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@osd.11.service' 2024-04-26T20:52:18.348 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:18.349 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-26T20:52:18.349 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-26T20:52:18.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.a 2024-04-26T20:52:18.898 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 26 20:52:18 smithi102 systemd[1]: Stopping Ceph mds.a for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:19.399 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 26 20:52:18 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-a[69497]: 2024-04-26T20:52:18.910+0000 7fcf76a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:19.399 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 26 20:52:18 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-a[69497]: 2024-04-26T20:52:18.910+0000 7fcf76a00700 -1 mds.a *** got signal Terminated *** 2024-04-26T20:52:21.537 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 26 20:52:21 smithi102 podman[86692]: 2024-04-26 20:52:21.209810106 +0000 UTC m=+2.428121599 container died 1d74e291a0ea9a15b1317fb0f60a65ba1d9c554ebe2ead518d31772c6da36fe5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T20:52:21.889 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 26 20:52:21 smithi102 podman[86692]: 2024-04-26 20:52:21.654821822 +0000 UTC m=+2.873133313 container cleanup 1d74e291a0ea9a15b1317fb0f60a65ba1d9c554ebe2ead518d31772c6da36fe5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-26T20:52:21.890 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 26 20:52:21 smithi102 bash[86692]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-a 2024-04-26T20:52:22.148 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 26 20:52:21 smithi102 podman[86706]: 2024-04-26 20:52:21.888866514 +0000 UTC m=+0.663055867 container remove 1d74e291a0ea9a15b1317fb0f60a65ba1d9c554ebe2ead518d31772c6da36fe5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T20:52:22.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.a.service' 2024-04-26T20:52:22.519 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:22.519 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-26T20:52:22.519 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-26T20:52:22.519 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.d 2024-04-26T20:52:22.898 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:52:22 smithi102 systemd[1]: Stopping Ceph mds.d for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:23.398 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:52:23 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d[68484]: 2024-04-26T20:52:23.030+0000 7f0f37a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:23.399 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:52:23 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d[68484]: 2024-04-26T20:52:23.030+0000 7f0f37a00700 -1 mds.d *** got signal Terminated *** 2024-04-26T20:52:24.512 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:52:24 smithi102 podman[86811]: 2024-04-26 20:52:24.237614608 +0000 UTC m=+1.327663328 container died de3b9911ca8d39ca15e53834749d8ea9962b25aacf5b2d126f6b0f78318fbff3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-26T20:52:24.810 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:52:24 smithi102 podman[86811]: 2024-04-26 20:52:24.624927126 +0000 UTC m=+1.714975875 container cleanup de3b9911ca8d39ca15e53834749d8ea9962b25aacf5b2d126f6b0f78318fbff3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:52:24.810 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:52:24 smithi102 bash[86811]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d 2024-04-26T20:52:25.148 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 26 20:52:24 smithi102 podman[86825]: 2024-04-26 20:52:24.809149818 +0000 UTC m=+0.558790385 container remove de3b9911ca8d39ca15e53834749d8ea9962b25aacf5b2d126f6b0f78318fbff3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T20:52:25.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.d.service' 2024-04-26T20:52:25.417 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:25.417 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-26T20:52:25.417 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-26T20:52:25.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.g 2024-04-26T20:52:25.898 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:52:25 smithi102 systemd[1]: Stopping Ceph mds.g for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:26.398 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:52:25 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g[69117]: 2024-04-26T20:52:25.991+0000 7f26fd200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:26.399 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:52:25 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g[69117]: 2024-04-26T20:52:25.991+0000 7f26fd200700 -1 mds.g *** got signal Terminated *** 2024-04-26T20:52:28.525 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:52:28 smithi102 podman[86931]: 2024-04-26 20:52:28.246190363 +0000 UTC m=+2.417990912 container died facab8299a722fec42fa8b44cfbde018c89c74c8240fed0ec2295e660cb46dd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:52:28.888 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:52:28 smithi102 podman[86931]: 2024-04-26 20:52:28.686440111 +0000 UTC m=+2.858240656 container cleanup facab8299a722fec42fa8b44cfbde018c89c74c8240fed0ec2295e660cb46dd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T20:52:28.888 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:52:28 smithi102 bash[86931]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g 2024-04-26T20:52:29.148 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 26 20:52:28 smithi102 podman[86945]: 2024-04-26 20:52:28.8876193 +0000 UTC m=+0.627313753 container remove facab8299a722fec42fa8b44cfbde018c89c74c8240fed0ec2295e660cb46dd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T20:52:29.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.g.service' 2024-04-26T20:52:29.505 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:29.505 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-26T20:52:29.506 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-26T20:52:29.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.j 2024-04-26T20:52:30.131 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:52:29 smithi102 systemd[1]: Stopping Ceph mds.j for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:30.398 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:52:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-j[68122]: 2024-04-26T20:52:30.129+0000 7ffb6ee00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:30.399 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:52:30 smithi102 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-j[68122]: 2024-04-26T20:52:30.129+0000 7ffb6ee00700 -1 mds.j *** got signal Terminated *** 2024-04-26T20:52:34.398 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:52:34 smithi102 podman[87050]: 2024-04-26 20:52:34.136855152 +0000 UTC m=+4.129785600 container died 0badd3389a910845252705c1113280712e069a0a1062347d9dda2d273c08324d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T20:52:34.708 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:52:34 smithi102 podman[87050]: 2024-04-26 20:52:34.523562176 +0000 UTC m=+4.516492643 container cleanup 0badd3389a910845252705c1113280712e069a0a1062347d9dda2d273c08324d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T20:52:34.709 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:52:34 smithi102 bash[87050]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-j 2024-04-26T20:52:35.148 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 26 20:52:34 smithi102 podman[87064]: 2024-04-26 20:52:34.707510531 +0000 UTC m=+0.549295738 container remove 0badd3389a910845252705c1113280712e069a0a1062347d9dda2d273c08324d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T20:52:35.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.j.service' 2024-04-26T20:52:35.290 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:35.290 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-26T20:52:35.290 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-26T20:52:35.290 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.b 2024-04-26T20:52:35.581 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:52:35 smithi112 systemd[1]: Stopping Ceph mds.b for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:35.839 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:52:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b[62857]: 2024-04-26T20:52:35.581+0000 7ff62addc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:35.839 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:52:35 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b[62857]: 2024-04-26T20:52:35.581+0000 7ff62addc700 -1 mds.b *** got signal Terminated *** 2024-04-26T20:52:39.923 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:52:39 smithi112 podman[68344]: 2024-04-26 20:52:39.590040227 +0000 UTC m=+4.114908296 container died bce7f06a9fee546b2f861d5137a54cdd4b7a1c4ccba8cd4dd563bca9dcc3d8d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T20:52:40.232 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:52:40 smithi112 podman[68344]: 2024-04-26 20:52:40.005838172 +0000 UTC m=+4.530706239 container cleanup bce7f06a9fee546b2f861d5137a54cdd4b7a1c4ccba8cd4dd563bca9dcc3d8d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-26T20:52:40.232 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:52:40 smithi112 bash[68344]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b 2024-04-26T20:52:40.482 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.b.service' 2024-04-26T20:52:40.503 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:52:40 smithi112 podman[68370]: 2024-04-26 20:52:40.232178292 +0000 UTC m=+0.638940191 container remove bce7f06a9fee546b2f861d5137a54cdd4b7a1c4ccba8cd4dd563bca9dcc3d8d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T20:52:40.503 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:52:40 smithi112 systemd[1]: ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.b.service: Deactivated successfully. 2024-04-26T20:52:40.504 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:52:40 smithi112 systemd[1]: Stopped Ceph mds.b for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:52:40.504 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 26 20:52:40 smithi112 systemd[1]: ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.b.service: Consumed 7.095s CPU time. 2024-04-26T20:52:40.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:40.516 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-26T20:52:40.517 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-26T20:52:40.517 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.e 2024-04-26T20:52:40.776 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:52:40 smithi112 systemd[1]: Stopping Ceph mds.e for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:41.276 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:52:40 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e[62218]: 2024-04-26T20:52:40.868+0000 7fb5de717700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:41.276 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:52:40 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e[62218]: 2024-04-26T20:52:40.868+0000 7fb5de717700 -1 mds.e *** got signal Terminated *** 2024-04-26T20:52:44.444 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:52:44 smithi112 podman[68478]: 2024-04-26 20:52:44.188437973 +0000 UTC m=+3.454255590 container died 29e7af1817b9d3a98bb600c6802ff9062b3a1cedb01b6d0dc538c8918edc37e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T20:52:44.776 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:52:44 smithi112 podman[68478]: 2024-04-26 20:52:44.565765102 +0000 UTC m=+3.831582706 container cleanup 29e7af1817b9d3a98bb600c6802ff9062b3a1cedb01b6d0dc538c8918edc37e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T20:52:44.777 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 26 20:52:44 smithi112 bash[68478]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-e 2024-04-26T20:52:45.044 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.e.service' 2024-04-26T20:52:45.076 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:45.076 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-26T20:52:45.076 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-26T20:52:45.076 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.h 2024-04-26T20:52:45.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:52:45 smithi112 systemd[1]: Stopping Ceph mds.h for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:45.776 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:52:45 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h[62534]: 2024-04-26T20:52:45.405+0000 7f907cfe4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:45.776 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:52:45 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h[62534]: 2024-04-26T20:52:45.405+0000 7f907cfe4700 -1 mds.h *** got signal Terminated *** 2024-04-26T20:52:48.447 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:52:48 smithi112 podman[68600]: 2024-04-26 20:52:48.181571668 +0000 UTC m=+2.886232978 container died 0d63dc159fe08018cffad509ae11467c7f684c974b407a8a0a6d4f4949cdc1d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True) 2024-04-26T20:52:48.776 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:52:48 smithi112 podman[68600]: 2024-04-26 20:52:48.546850882 +0000 UTC m=+3.251512193 container cleanup 0d63dc159fe08018cffad509ae11467c7f684c974b407a8a0a6d4f4949cdc1d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:52:48.776 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 26 20:52:48 smithi112 bash[68600]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-h 2024-04-26T20:52:48.964 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.h.service' 2024-04-26T20:52:48.995 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:48.995 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-26T20:52:48.995 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-26T20:52:48.996 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.k 2024-04-26T20:52:49.342 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:52:49 smithi112 systemd[1]: Stopping Ceph mds.k for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:49.776 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:52:49 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-k[61899]: 2024-04-26T20:52:49.342+0000 7f47855d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:49.776 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:52:49 smithi112 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-k[61899]: 2024-04-26T20:52:49.342+0000 7f47855d1700 -1 mds.k *** got signal Terminated *** 2024-04-26T20:52:52.435 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:52:52 smithi112 podman[68722]: 2024-04-26 20:52:52.170641721 +0000 UTC m=+2.962072011 container died 1867ebd92353c40880ef9ba548e5c1319e95caab9c6052cfbdbaaa53eea465ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T20:52:52.710 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:52:52 smithi112 podman[68722]: 2024-04-26 20:52:52.517146951 +0000 UTC m=+3.308577241 container cleanup 1867ebd92353c40880ef9ba548e5c1319e95caab9c6052cfbdbaaa53eea465ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:52:52.710 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:52:52 smithi112 bash[68722]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-k 2024-04-26T20:52:52.937 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.k.service' 2024-04-26T20:52:52.973 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:52:52 smithi112 podman[68736]: 2024-04-26 20:52:52.709971795 +0000 UTC m=+0.533568425 container remove 1867ebd92353c40880ef9ba548e5c1319e95caab9c6052cfbdbaaa53eea465ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T20:52:52.974 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:52:52 smithi112 systemd[1]: ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.k.service: Deactivated successfully. 2024-04-26T20:52:52.974 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:52:52 smithi112 systemd[1]: Stopped Ceph mds.k for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:52:52.974 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 26 20:52:52 smithi112 systemd[1]: ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.k.service: Consumed 2.504s CPU time. 2024-04-26T20:52:52.983 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:52.984 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-26T20:52:52.984 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-26T20:52:52.984 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.c 2024-04-26T20:52:53.371 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:52:53 smithi130 systemd[1]: Stopping Ceph mds.c for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:53.372 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:52:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c[63351]: 2024-04-26T20:52:53.253+0000 7fd840a3b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:53.372 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:52:53 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c[63351]: 2024-04-26T20:52:53.253+0000 7fd840a3b700 -1 mds.c *** got signal Terminated *** 2024-04-26T20:52:56.495 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:52:56 smithi130 podman[69175]: 2024-04-26 20:52:56.18023938 +0000 UTC m=+3.012681452 container died dd597cbb22147206c39303027f1c061341cafc8c1b598ef26405edc131b2f93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T20:52:56.495 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:52:56 smithi130 podman[69175]: 2024-04-26 20:52:56.495152545 +0000 UTC m=+3.327594614 container cleanup dd597cbb22147206c39303027f1c061341cafc8c1b598ef26405edc131b2f93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T20:52:56.871 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:52:56 smithi130 bash[69175]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c 2024-04-26T20:52:56.871 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 26 20:52:56 smithi130 podman[69201]: 2024-04-26 20:52:56.704718171 +0000 UTC m=+0.527439532 container remove dd597cbb22147206c39303027f1c061341cafc8c1b598ef26405edc131b2f93a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T20:52:56.914 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.c.service' 2024-04-26T20:52:56.949 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:52:56.950 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-26T20:52:56.950 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-26T20:52:56.950 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.f 2024-04-26T20:52:57.275 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:52:57 smithi130 systemd[1]: Stopping Ceph mds.f for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:52:57.621 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:52:57 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f[63037]: 2024-04-26T20:52:57.273+0000 7fc78c875700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:52:57.622 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:52:57 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f[63037]: 2024-04-26T20:52:57.273+0000 7fc78c875700 -1 mds.f *** got signal Terminated *** 2024-04-26T20:53:00.621 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:53:00 smithi130 podman[69307]: 2024-04-26 20:53:00.174066932 +0000 UTC m=+3.014467169 container died d3e425d7e51f3b00ce6e139ba53bea43a344540dc669783446d023a4df3b7599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-26T20:53:01.025 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:53:00 smithi130 podman[69307]: 2024-04-26 20:53:00.757823296 +0000 UTC m=+3.598223527 container cleanup d3e425d7e51f3b00ce6e139ba53bea43a344540dc669783446d023a4df3b7599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T20:53:01.025 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 26 20:53:00 smithi130 bash[69307]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-f 2024-04-26T20:53:01.181 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.f.service' 2024-04-26T20:53:01.212 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:53:01.212 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-26T20:53:01.212 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-26T20:53:01.213 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.i 2024-04-26T20:53:01.621 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:53:01 smithi130 systemd[1]: Stopping Ceph mds.i for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:53:01.621 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:53:01 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i[63662]: 2024-04-26T20:53:01.536+0000 7f1758fa4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:53:01.622 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:53:01 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i[63662]: 2024-04-26T20:53:01.536+0000 7f1758fa4700 -1 mds.i *** got signal Terminated *** 2024-04-26T20:53:04.621 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:53:04 smithi130 podman[69427]: 2024-04-26 20:53:04.179292812 +0000 UTC m=+2.744261436 container died 140f6230e432d7b006a3c585cbc450248d40cfbe47d0284ca63d8de99e93fafe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T20:53:04.912 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:53:04 smithi130 podman[69427]: 2024-04-26 20:53:04.719739124 +0000 UTC m=+3.284707755 container cleanup 140f6230e432d7b006a3c585cbc450248d40cfbe47d0284ca63d8de99e93fafe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T20:53:04.912 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:53:04 smithi130 bash[69427]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i 2024-04-26T20:53:05.151 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.i.service' 2024-04-26T20:53:05.170 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:53:04 smithi130 podman[69441]: 2024-04-26 20:53:04.91243443 +0000 UTC m=+0.733142372 container remove 140f6230e432d7b006a3c585cbc450248d40cfbe47d0284ca63d8de99e93fafe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T20:53:05.171 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:53:05 smithi130 systemd[1]: ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.i.service: Deactivated successfully. 2024-04-26T20:53:05.171 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:53:05 smithi130 systemd[1]: Stopped Ceph mds.i for a94c830a-040a-11ef-bc93-c7b262605968. 2024-04-26T20:53:05.171 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 26 20:53:05 smithi130 systemd[1]: ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.i.service: Consumed 2.133s CPU time. 2024-04-26T20:53:05.181 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:53:05.182 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-26T20:53:05.182 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-26T20:53:05.182 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.l 2024-04-26T20:53:05.539 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:53:05 smithi130 systemd[1]: Stopping Ceph mds.l for a94c830a-040a-11ef-bc93-c7b262605968... 2024-04-26T20:53:05.871 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:53:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l[62726]: 2024-04-26T20:53:05.539+0000 7f32c679d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T20:53:05.872 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:53:05 smithi130 ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l[62726]: 2024-04-26T20:53:05.539+0000 7f32c679d700 -1 mds.l *** got signal Terminated *** 2024-04-26T20:53:08.621 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:53:08 smithi130 podman[69546]: 2024-04-26 20:53:08.17609153 +0000 UTC m=+2.770881007 container died 5e5f8f74aad5b81fe5d7ca91018c2bace7ffb62cbc8d056101f366eb1fa766ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T20:53:09.116 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:53:08 smithi130 podman[69546]: 2024-04-26 20:53:08.839864136 +0000 UTC m=+3.434653607 container cleanup 5e5f8f74aad5b81fe5d7ca91018c2bace7ffb62cbc8d056101f366eb1fa766ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T20:53:09.116 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 26 20:53:08 smithi130 bash[69546]: ceph-a94c830a-040a-11ef-bc93-c7b262605968-mds-l 2024-04-26T20:53:09.259 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a94c830a-040a-11ef-bc93-c7b262605968@mds.l.service' 2024-04-26T20:53:09.289 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T20:53:09.289 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-26T20:53:09.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a94c830a-040a-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T20:53:09.616 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:53:14.765 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a94c830a-040a-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T20:53:14.961 INFO:teuthology.orchestra.run.smithi112.stdout:Deleting cluster with fsid: a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:53:17.324 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a94c830a-040a-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T20:53:17.513 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:53:19.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:53:19.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:53:19.918 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T20:53:19.944 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T20:53:19.946 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159/remote/smithi102/crash 2024-04-26T20:53:19.947 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/crash -- . 2024-04-26T20:53:20.044 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T20:53:20.044 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2024-04-26T20:53:20.049 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159/remote/smithi112/crash 2024-04-26T20:53:20.049 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/crash -- . 2024-04-26T20:53:20.084 INFO:teuthology.orchestra.run.smithi112.stderr:tar: /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T20:53:20.085 INFO:teuthology.orchestra.run.smithi112.stderr:tar: Error is not recoverable: exiting now 2024-04-26T20:53:20.086 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159/remote/smithi130/crash 2024-04-26T20:53:20.087 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/crash -- . 2024-04-26T20:53:20.124 INFO:teuthology.orchestra.run.smithi130.stderr:tar: /var/lib/ceph/a94c830a-040a-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T20:53:20.124 INFO:teuthology.orchestra.run.smithi130.stderr:tar: Error is not recoverable: exiting now 2024-04-26T20:53:20.125 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T20:53:20.126 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-26T20:53:20.231 INFO:tasks.cephadm:Compressing logs... 2024-04-26T20:53:20.231 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T20:53:20.235 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T20:53:20.238 DEBUG:teuthology.orchestra.run.smithi130:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T20:53:20.260 INFO:teuthology.orchestra.run.smithi130.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T20:53:20.261 INFO:teuthology.orchestra.run.smithi130.stderr:: No such file or directory 2024-04-26T20:53:20.261 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.log 2024-04-26T20:53:20.261 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T20:53:20.264 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-26T20:53:20.264 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T20:53:20.264 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-26T20:53:20.264 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.audit.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T20:53:20.264 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-26T20:53:20.264 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-26T20:53:20.265 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-26T20:53:20.265 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-26T20:53:20.266 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mgr.z.log: 88.7% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T20:53:20.266 INFO:teuthology.orchestra.run.smithi130.stderr: 90.5% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T20:53:20.267 INFO:teuthology.orchestra.run.smithi112.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory 2024-04-26T20:53:20.267 INFO:teuthology.orchestra.run.smithi112.stderr: /var/log/ceph/cephadm.log 2024-04-26T20:53:20.267 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.log 2024-04-26T20:53:20.267 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T20:53:20.267 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-26T20:53:20.268 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-26T20:53:20.271 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-26T20:53:20.271 INFO:teuthology.orchestra.run.smithi112.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T20:53:20.272 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T20:53:20.272 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-26T20:53:20.272 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-26T20:53:20.272 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-26T20:53:20.272 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T20:53:20.272 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- 90.4% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T20:53:20.272 INFO:teuthology.orchestra.run.smithi112.stderr: /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T20:53:20.272 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-26T20:53:20.273 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.6% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T20:53:20.275 INFO:teuthology.orchestra.run.smithi130.stderr: 93.2% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-26T20:53:20.276 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T20:53:20.276 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-26T20:53:20.276 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-26T20:53:20.276 INFO:teuthology.orchestra.run.smithi130.stderr: 82.5% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T20:53:20.276 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T20:53:20.276 INFO:teuthology.orchestra.run.smithi112.stderr: 93.1% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-26T20:53:20.283 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T20:53:20.283 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-26T20:53:20.286 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-26T20:53:20.294 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’gzip: No such file or directory 2024-04-26T20:53:20.295 INFO:teuthology.orchestra.run.smithi102.stderr: -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T20:53:20.296 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-26T20:53:20.296 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T20:53:20.297 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.log 2024-04-26T20:53:20.297 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T20:53:20.298 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-26T20:53:20.298 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-26T20:53:20.299 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.g.log: 88.7% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T20:53:20.300 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-26T20:53:20.301 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T20:53:20.302 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.3.log: 90.3% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T20:53:20.302 INFO:teuthology.orchestra.run.smithi102.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T20:53:20.303 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-26T20:53:20.303 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T20:53:20.304 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.0.log: /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-26T20:53:20.306 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T20:53:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.6.log: /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.8% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T20:53:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:gzip 94.5% -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-26T20:53:20.310 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-26T20:53:20.312 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T20:53:20.313 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T20:53:20.317 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-26T20:53:20.318 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-26T20:53:20.320 INFO:teuthology.orchestra.run.smithi130.stderr: 94.1% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-26T20:53:20.323 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.d.log: /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-26T20:53:20.328 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.log: 94.6% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-26T20:53:20.329 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.0% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-26T20:53:20.331 INFO:teuthology.orchestra.run.smithi130.stderr: 94.1% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-26T20:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr: 90.9% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T20:53:20.334 INFO:teuthology.orchestra.run.smithi102.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-26T20:53:20.354 INFO:teuthology.orchestra.run.smithi112.stderr: 94.6% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-26T20:53:20.358 INFO:teuthology.orchestra.run.smithi130.stderr: 94.5% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-26T20:53:20.361 INFO:teuthology.orchestra.run.smithi102.stderr: 94.1% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-26T20:53:20.379 INFO:teuthology.orchestra.run.smithi130.stderr: 94.6% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-26T20:53:20.387 INFO:teuthology.orchestra.run.smithi112.stderr: 94.6% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-26T20:53:20.405 INFO:teuthology.orchestra.run.smithi102.stderr: 94.4% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-26T20:53:20.477 INFO:teuthology.orchestra.run.smithi112.stderr: 93.7% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-26T20:53:20.681 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-26T20:53:20.684 INFO:teuthology.orchestra.run.smithi102.stderr: 89.9% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-26T20:53:20.819 INFO:teuthology.orchestra.run.smithi130.stderr: 92.0% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-26T20:53:21.062 INFO:teuthology.orchestra.run.smithi130.stderr: 92.5% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T20:53:21.121 INFO:teuthology.orchestra.run.smithi112.stderr: 92.5% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-26T20:53:21.140 INFO:teuthology.orchestra.run.smithi130.stderr: 92.6% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-26T20:53:21.194 INFO:teuthology.orchestra.run.smithi102.stderr: 92.3% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-26T20:53:21.197 INFO:teuthology.orchestra.run.smithi112.stderr: 92.2% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T20:53:21.255 INFO:teuthology.orchestra.run.smithi130.stderr: 92.3% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-26T20:53:21.321 INFO:teuthology.orchestra.run.smithi102.stderr: 90.5% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T20:53:21.374 INFO:teuthology.orchestra.run.smithi102.stderr: 92.6% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T20:53:21.395 INFO:teuthology.orchestra.run.smithi102.stderr: 92.6% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-26T20:53:21.429 INFO:teuthology.orchestra.run.smithi112.stderr: 92.5% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-26T20:53:21.470 INFO:teuthology.orchestra.run.smithi130.stderr: 93.5% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-26T20:53:21.472 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-04-26T20:53:21.472 INFO:teuthology.orchestra.run.smithi130.stderr:real 0m1.220s 2024-04-26T20:53:21.472 INFO:teuthology.orchestra.run.smithi130.stderr:user 0m4.556s 2024-04-26T20:53:21.472 INFO:teuthology.orchestra.run.smithi130.stderr:sys 0m0.163s 2024-04-26T20:53:21.696 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-26T20:53:21.697 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-04-26T20:53:21.698 INFO:teuthology.orchestra.run.smithi112.stderr:real 0m1.445s 2024-04-26T20:53:21.698 INFO:teuthology.orchestra.run.smithi112.stderr:user 0m5.008s 2024-04-26T20:53:21.698 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m0.209s 2024-04-26T20:53:21.771 INFO:teuthology.orchestra.run.smithi102.stderr: 93.1% -- replaced with /var/log/ceph/a94c830a-040a-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-26T20:53:21.775 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-04-26T20:53:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m1.507s 2024-04-26T20:53:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m5.603s 2024-04-26T20:53:21.775 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m0.544s 2024-04-26T20:53:21.776 INFO:tasks.cephadm:Archiving logs... 2024-04-26T20:53:21.776 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159/remote/smithi102/log 2024-04-26T20:53:21.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T20:53:22.610 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159/remote/smithi112/log 2024-04-26T20:53:22.611 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T20:53:23.352 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159/remote/smithi130/log 2024-04-26T20:53:23.353 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T20:53:24.020 INFO:tasks.cephadm:Removing cluster... 2024-04-26T20:53:24.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a94c830a-040a-11ef-bc93-c7b262605968 --force 2024-04-26T20:53:24.325 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:53:24.824 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a94c830a-040a-11ef-bc93-c7b262605968 --force 2024-04-26T20:53:25.016 INFO:teuthology.orchestra.run.smithi112.stdout:Deleting cluster with fsid: a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:53:25.266 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a94c830a-040a-11ef-bc93-c7b262605968 --force 2024-04-26T20:53:25.445 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: a94c830a-040a-11ef-bc93-c7b262605968 2024-04-26T20:53:25.706 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T20:53:25.707 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T20:53:25.764 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T20:53:25.780 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T20:53:25.795 INFO:tasks.cephadm:Teardown complete 2024-04-26T20:53:25.796 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T20:53:25.807 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T20:53:25.807 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T20:53:25.814 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T20:53:25.822 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T20:53:25.966 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T20:53:25.979 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T20:53:25.979 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T20:53:25.987 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T20:53:25.990 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T20:53:26.001 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-04-26T20:53:26.003 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-04-26T20:53:26.006 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T20:53:26.006 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-04-26T20:53:26.006 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv01.front.sepia.ceph.com 3 6 377 40 +1255us[+1267us] +/- 51ms 2024-04-26T20:53:26.006 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 106 -1037us[-1014us] +/- 62ms 2024-04-26T20:53:26.006 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 -655us[ -655us] +/- 76ms 2024-04-26T20:53:26.006 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:53:26.007 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T20:53:26.007 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-04-26T20:53:26.007 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 37 +1257us[+1262us] +/- 51ms 2024-04-26T20:53:26.008 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv02.front.sepia.ceph.com 3 7 377 37 -1005us[-1000us] +/- 63ms 2024-04-26T20:53:26.008 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 -641us[ -636us] +/- 76ms 2024-04-26T20:53:26.008 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:53:26.033 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-04-26T20:53:26.043 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T20:53:26.043 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-04-26T20:53:26.043 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv01.front.sepia.ceph.com 3 6 377 36 +1271us[+1255us] +/- 51ms 2024-04-26T20:53:26.044 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -1042us[-1042us] +/- 63ms 2024-04-26T20:53:26.044 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 -644us[ -660us] +/- 76ms 2024-04-26T20:53:26.044 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T20:53:26.045 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T20:53:26.055 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T20:53:26.055 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T20:53:26.065 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T20:53:26.251 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T20:53:26.319 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T20:53:26.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T20:53:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:53:26.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T20:53:26.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:53:26.505 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T20:53:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:53:26.531 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T20:53:26.541 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T20:53:26.556 INFO:teuthology.task.internal:Duration was 2348.936554 seconds 2024-04-26T20:53:26.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T20:53:26.602 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T20:53:26.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T20:53:26.611 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T20:53:26.613 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T20:53:26.647 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T20:53:26.650 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T20:53:26.750 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T20:53:27.241 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T20:53:27.241 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-04-26T20:53:27.241 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T20:53:27.310 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-04-26T20:53:27.310 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T20:53:27.335 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2024-04-26T20:53:27.335 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T20:53:27.368 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T20:53:27.369 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T20:53:27.373 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T20:53:27.377 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T20:53:27.434 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T20:53:27.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T20:53:27.439 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T20:53:27.446 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T20:53:27.932 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T20:53:27.946 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T20:53:27.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T20:53:28.023 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T20:53:28.046 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T20:53:28.070 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T20:53:28.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:53:28.088 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:53:28.090 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:53:28.109 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-04-26T20:53:28.132 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2024-04-26T20:53:28.175 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-04-26T20:53:28.216 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:53:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:53:28.262 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:53:28.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:53:28.276 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T20:53:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T20:53:28.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T20:53:28.301 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T20:53:28.301 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159/remote/smithi102 2024-04-26T20:53:28.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T20:53:28.405 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159/remote/smithi112 2024-04-26T20:53:28.405 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T20:53:28.439 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159/remote/smithi130 2024-04-26T20:53:28.440 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T20:53:28.473 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T20:53:28.474 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T20:53:28.481 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T20:53:28.484 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T20:53:28.532 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T20:53:28.544 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T20:53:28.544 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T20:53:28.554 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T20:53:28.554 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T20:53:28.560 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T20:53:28.562 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T20:53:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 20:53 /home/ubuntu/cephtest 2024-04-26T20:53:28.576 INFO:teuthology.orchestra.run.smithi130.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 20:53 /home/ubuntu/cephtest 2024-04-26T20:53:28.607 INFO:teuthology.orchestra.run.smithi102.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 20:53 /home/ubuntu/cephtest 2024-04-26T20:53:28.610 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T20:53:28.621 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T20:53:28.645 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T20:53:28.672 DEBUG:teuthology.nuke:shortname: smithi102 2024-04-26T20:53:28.673 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T20:53:28.695 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 20:06:03.127248', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWvmtzbHbLFhJ97Cu/iv+xsO6C9RwzQEyhw12pI0j8348FADsk9O6WHmEVYqNJ5KG/VCJef3q9XUBl6PWeMwr8='} 2024-04-26T20:53:28.700 DEBUG:teuthology.nuke:shortname: smithi112 2024-04-26T20:53:28.700 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T20:53:28.720 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 20:06:03.130119', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMIaKD/jMdpOkfA1Ox+h+n8f6ujZhyxeF1Lrd4qG4U4yuoyGiSglOH20N5j+stq60Rkw8S3lkxe2EB3+742n4kQ='} 2024-04-26T20:53:28.723 DEBUG:teuthology.nuke:shortname: smithi130 2024-04-26T20:53:28.723 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T20:53:28.740 INFO:teuthology.orchestra.console.smithi102:Power off 2024-04-26T20:53:28.740 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T20:53:28.752 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675159', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 20:06:03.128825', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB5kOYp3W0wKwhvvFJ9sier57RU5hsuezJTJFqoHJPZt/gvPOVEs2SocAoB7qdgTKvhqTu8Q2kpF0xie1BU28pc='} 2024-04-26T20:53:28.763 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2024-04-26T20:53:28.763 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:53:28.784 INFO:teuthology.orchestra.console.smithi112:Power off 2024-04-26T20:53:28.784 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T20:53:28.799 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-04-26T20:53:28.808 DEBUG:teuthology.orchestra.console.smithi112:power off output: Chassis Power Control: Down/Off 2024-04-26T20:53:28.808 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:53:28.827 INFO:teuthology.orchestra.console.smithi130:Power off 2024-04-26T20:53:28.828 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T20:53:28.837 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-04-26T20:53:28.849 DEBUG:teuthology.orchestra.console.smithi130:power off output: Chassis Power Control: Down/Off 2024-04-26T20:53:28.849 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:53:28.874 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-04-26T20:53:32.800 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:53:32.838 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:53:32.875 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:53:32.911 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-04-26T20:53:32.949 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-04-26T20:53:32.984 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-04-26T20:53:36.914 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:53:36.950 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:53:36.985 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T20:53:37.024 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2024-04-26T20:53:37.061 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is off 2024-04-26T20:53:37.093 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-26T20:53:37.126 INFO:teuthology.orchestra.console.smithi102:Power off completed 2024-04-26T20:53:37.162 INFO:teuthology.orchestra.console.smithi112:Power off completed 2024-04-26T20:53:37.194 INFO:teuthology.orchestra.console.smithi130:Power off completed 2024-04-26T20:53:37.300 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} duration: 2348.9365541934967 failure_reason: 'Command failed on smithi102 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs qa random 0.10''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=aa14764582a04792bb7c86b55472e911 status: fail success: false 2024-04-26T20:53:37.301 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T20:53:37.377 INFO:teuthology.run:FAIL